freeCodeCamp/tools/dashboard/docs
Mrugesh Mohapatra 21b27cb7c1
feat: import contribute tools (#38509)
* first commit

* add: adding all files

* added package-lock.json

* fix: labler and getOpenPrs working

* fix: fixed bug with prFilter function

* feat: can check for multiple errors

* add: install more libraries

* refactor: uses getOpenPrs to get PRs

* packages: modified

* feature: added addComment feature

* feature: added ability to check guide errors

* add: added  guideFolderChecks

* fix: corrected issue with open/close

* update: installed new libraries

* fix: updated guide checker

* feat: create license

* fix: update the comment template (#2)

* fix: comment section

* fix: add org name keys

* fix: corrected guide error logic

* fix: removed console.log

Co-Authored-By: RandellDawson <5313213+RandellDawson@users.noreply.github.com>

* fix: corrected guideFolderChecks reduce

* refactor: removed setInterval logic

* feat: added new utilities module

* fix: addComment and labler issues

* feat: changed to module.exports

* fix: improved log file data

* fix: use config var for rateLimiter

Co-Authored-By: RandellDawson <5313213+RandellDawson@users.noreply.github.com>

* fix: used config var for rateLimiter

Co-Authored-By: RandellDawson <5313213+RandellDawson@users.noreply.github.com>

* fix: add vars for production

* fix: add work-logs config

* fix: update gitconfig for work-logs

* fix: update path to work-logs

* docs(readme): add exception for admin v. user

* fix: dotenv will coerce boolean from string (#10)

* fix: changed to sync file save

* feat: updated label script

* feat: installed new libraries

* feat: renamed labelOpenPrs.js to sweeper.js

* feat: 1st attempt repo restructure

* feat: added descriptions to one-off scripts

* fix: rewrote the missing frontmatter message

* feat: created one-off scripts to find frontmatter issues

* fix: add links to relevant style guide

* fix: corrected gitignore input-files

* feat: renamed files/folders to lowercase

* fix: changed prStat.js to pr-stats.js

* fix: renamed prStat.js to pr-stats.js

* fix: renamed addLabels.js to add-labels.js

* fix: deleted typo

* feat: changed .env variables

* fix: added rateLimiter after retrieving pr files

* feat: improved logging functionality

* fix: added catch for unexpected frontmatter error

* fix: changed verbiage unexpected frontmatter error

* fix: correct logging bug

* feat: improved findFailures.js and closeOpen.js

* feat: modified the close/open scripts

* refac: moved labeler.js to pr-tasks

* fix: changed rateLimter time in close-open

* fix: changed delay between close/open

* install: added cli-progress library

* feat: added 2 progress bars for sweeper

* fix: changed rateLimiter for pr-relations script

* fix: corrected directory path for renaming of file to work

* first commit

* first commit

* test travis setup

* more travis setup

* get payload

* test

* test

* test

* app yml pull_requests write

* non-working pullRequest createComment

* problem setting permissions

* create-react-app setup.

* Added styled-components dependency.

* Merge pull request #1 from RandellDawson/feature/prototype

Created prototype.

* Merge pull request #3 from RandellDawson/feat/add-links-to-results

fix: improved input elem behavior

* wip

* test on repo

* tests passing

* fix test title

* issues--write in app.yml

* fix getLabel to createLabel pipeline

* this github scope

* Merge pull request #5 from RandellDawson/refactor/input-react-component

Implemented ref forwarding for Input component. Minor clean up.

* Merge pull request #6 from RandellDawson/feature/search-with-enter-keypress

Added the ability to search by pressing Enter. Associated refactoring.

* Merge pull request #7 from RandellDawson/feature/display-username-with-prs

Added the ability for usernames to be displayed along with found PRs.

* Merge pull request #8 from RandellDawson/feat/report-improvements

Feature: Added Pareto Report and Tabs Navigation

* Merge pull request #9 from RandellDawson/feature/styled-tab-borders

Restyled tab borders to remove doubled-up borders.

* Merge pull request #10 from RandellDawson/feature/styled-components-theme

Implemented styled-components theme.

* Merge pull request #11 from RandellDawson/feature/root-package-json

Adde `package.json` and `.gitignore` to the root folder.

* Merge pull request #14 from RandellDawson/feat/create-search-component

[feature] created Search component

* Merge pull request #16 from RandellDawson/feat/create-search-component

[Feature] Finalized Search Component functionality

* Merge pull request #18 from RandellDawson/fix/correct-search-result-messages

[Fix] Correct search result messages and add info footer

* Merge pull request #19 from RandellDawson/fix/change-link-color

[Fix] Changed links from blue to 006400

* refactor: move sweeper logic to its own directory

* feat: lernafy the sweeper

* refactor: prepare for dashboard backend move

* docs: re-arrange files for cleaner structure

* refactor: prepare @tbushman 's probot for move

* fix: prepare for importing probot

* chore(.gitignore): update root level and remove redundant copy

* refactor: prepare dashboard-client for move

* refactor: prepare repo for bringing in dashboard client

* chore: install deps, remove clutter

* feat: improve efficiency of pr relations script (#23)

* chore: update lock file (#24)

* Feature: API endpoint constants (#30)

* Implemented expeced behaviour for text input.

* Added constants for API endpoints. Refactored Footer, Pareto and Search components accordingly.

* fix: fixed several issues after move to mono repo (#31)

This PR has the following items:

1. Introduces a couple of fixes to the sweeper script from issues right before the move to the mono repo.  These were related to moving labeler out of the root of the sweeper folder and placing into the pr-tasks sub-folder.

2. Combined two scripts which were being used to update the data manually on Glitch (pr-relations.glitch.me).  The new one-off script (get-pr-relations-data.js) downloads the current data.json and then pulls down applicable Github data which updates the data.json and automatically uploads it back to the Glitch server.

3. In an effort to use the same log file across all current one-off scripts and sweeper.js, the ProcessLog class was modified, so now every log created has the same JSON structure as is uploaded to the Glitch server.  This removed a lot of redundant code across 3 files.

4. During a pair-coding session with @honmanyau, we tweaked the UI for the Dashboard app to give it a more consistent look across all views.

5. Based on feedback from a couple of the moderators using the new Dashboard app, the PR title was added along with some other styling. 

6. Added some environment variables for running dashboard-api and dashboard-client in a development mode.

7.  Added a sample_data.json file as a starter file, so someone does not have to do a full data pull to get things up and running. 

8. Modified Pareto view to only display files with 2 or more PRs to speed up the report.

* fix(.env): update paths relative from root of repo (#37)

* docs: update notes on the setup

* feat(tools): add eslint and prettier (#19)

* refactor(lint): apply linting from the tools (#39)

* Revert "refactor(lint): apply linting from the tools (#39)" (#40)

This reverts commit 8be7d12cb3.

* chore: made changes to move off of Glitch

* fix: removed extra lines from sample.env

* docs: added local setup instructions

* docs: update local setup instruction

* docs: update contributing

* docs: create readme.md

* fix(client): CRA config conflicts with the babel-eslint from root (#42)

* feat: add routes and mongodb (#34)

* fix: removed dependace on data.json

* docs: updated README.md

* fix: corrected issues with pareto.js and pr.js

* refactor: simplified mongoose related logic

* feat: added seed shortcut to package.json

* fix: corrected typo

* fix: fixed bug with info.js

* fix: simplified info.js

* fix: corrected updatedAt Schema issue

* fix: corrected typo in comment

* chore(lockfile): update after install from root

* fix: add a seed script to root

* Fix/restructuring and unify code logic (#48)

* fix: first pass at restructure linting cleanup

* fix: second pass of restructure

* fix: corrected path for index.html

* Fix/restructuring (#49)

* moving server scripts to probot/index.js

* non-working express.static config

* config move for seed

* change constant API_HOST to /contribute/

* remove winston

* remove error middleware

* put back client in lerna.json

* fix: fixed CRA static folder issue

* fix: added work-logs folder for one-offs

* small changes

* change /contribute route to /dashboard

* fix: use cross-env in package.json

* formatted with prettier (#50)

* encode mongo uri (#51)

* fix: removed ref to probot/client/.env

* Fixed the format command (#55)

* chore: update lockfile (#56)

* fix: corrected config.oneoff.productionRun issue (#52)

In an earlier PR I use `joi` to make PRODUCTION_RUN environment variables a Boolean, but did not change to the correct reference and did not change the code logic to compare to a Boolean instead of a String value (as it had been before).  This PR corrects that logic, so that the one-off scripts will work in a production mode again.  Also, removed the now unnecessary ` SKIP_PREFLIGHT_CHECK` environment variable from the `probot/.env` file.

* fix: ran npm run format

* fix: corrected some require paths for update-db.js

* fix: drop only specified collections in models.js

* fix: removed lerna for two run scripts

* remove unnecessary environment variables and npm scripts (#59)

* fix: removed config variables not needed

* fix: added pem files to gitignore

* feat: added one-off-script to add english labels

* fix: modified script description

* fix: updated script to use aggregate function

* fix: added note about needing to change url

* init tests

* .toHaveBeenCalledTimes(0) for unrelated PR files

* run tests from root

* lerna run test --scope determined by 'name' field in each package.json

* run jsx in node with .mjs

* shut down mongoose afterEach test

* npm-run-all

* mongoose in tests

* comment out failing script for now

* use probot to get filenames

* remove jsdom and uncomment mongo updateOne test

* only connect mongoose once and better test for PRtest.delete

* fix: added cross-env on npm test and start

* fix: ignore PEM files

* fix: get probot to add stff to db

* fix: changed to aggregate to get info

* fix: renamed update-db to reflect probot

* dashboard route working

* fix: remove calls to update db for now

* fix: update db script for crontab

* Fix: Removal of seed-db (#64)

* fix: removed seeding from production

* fix: removed npm start from instructions

* Fix: Simplify log files for one off scripts (#65)

* fix: removed seeding from production

* fix: removed npm start from instructions

* fix: removed indices from log file

* fix: update lockfile (#66)

* docs: create a CoC

* fix: increased the number of files pulled per PR to Github max of 300 (#70)

* fix: made CRA work with local server running

* added run command just for CRA dev mode

* feat: started working on adding Pareto filters

* fix: added missing fonts for landing page

* fix: filter functionality complete

* fix: removed other as an option

* fix: make language all if no language for filetype

* feat: add some develop scripts

* responsive css (#73)

* Fix broke link of contribution guide (#75)

* [Fix] Update landing page links and make styling for landing page and Dashboard match (#79)

* fix: corrected links and changed header links

* fix: added fonts

* fix: changed layout of Pareto options

* fix: match main page styling

* fix: adjusted padding on menu

* fix: correct link href for Home

* fix: changed title of Dashbaord

* [Fix] Corrected client page title typo (#83)

* fix: corrected typo in page title

* fix: add info record if no collection exists

* docs: updated docs relating to local dev ports

* fix: add count increment when adding

* fix: be able to specify lanuage

* feat: new script to find merge conflicts

* fix: added recheck for unknown status

* fix: add pr number to no results message

* fix: changed from notifying to adding label

* chore: update lockfiles (#92)

* docs: update setup docs (#93)

* fix: update lerna config to install in ci mode

* docs: update readme

* fix/update-local-setup-doc (#98)

* fix/update-local-setup-doc

* fix/capitalization

* Update docs/README.md

Co-Authored-By: Randell Dawson <5313213+RandellDawson@users.noreply.github.com>

* Update docs/README.md

Co-Authored-By: Randell Dawson <5313213+RandellDawson@users.noreply.github.com>

* Update docs/README.md

Co-Authored-By: Randell Dawson <5313213+RandellDawson@users.noreply.github.com>

* Update docs/README.md

Co-Authored-By: Randell Dawson <5313213+RandellDawson@users.noreply.github.com>

* fix/update-homepage-instruction

* fix: remove all guide related code

* chore: remove eslint config from dashboard

* fix: temporarily pause linting of tools/dashboard

Remove this when we have time to configure it properly.

* fix: pause testing on tools/dashboard

Co-authored-by: Randell Dawson <rdawson@onepathtech.com>
Co-authored-by: RandellDawson <5313213+RandellDawson@users.noreply.github.com>
Co-authored-by: tbushman <tbushman@pu.bli.sh>
Co-authored-by: Honman Yau <honmanyau@users.noreply.github.com>
Co-authored-by: Valeriy <ValeraS@users.noreply.github.com>
Co-authored-by: Tracey Bushman <tracey.bushman@gmail.com>
Co-authored-by: Niraj Nandish <nirajnandish@icloud.com>
Co-authored-by: Musthaq Ahamad <musthu.gm@gmail.com>
Co-authored-by: Tom <20648924+moT01@users.noreply.github.com>
Co-authored-by: Oliver Eyton-Williams <ojeytonwilliams@gmail.com>
2020-04-07 18:28:56 +05:30
..
CONTRIBUTING.md feat: import contribute tools (#38509) 2020-04-07 18:28:56 +05:30
README.md feat: import contribute tools (#38509) 2020-04-07 18:28:56 +05:30

README.md

Local Setup

  • Follow the steps below to get this running on your local machine

1. Copy .env

  • Copy the sample.env file into .env. The command below will do that in the terminal if your CWD(current working directory) is the contribute folder.
cp probot/sample.env probot/.env
  • If you do not want to populate the database with the freeCodeCamp PR's you can skip to step 6

2. Update .env

  • Put your GitHub username in the GITHUB_USERNAME field of the .env file

3. Obtain GitHub Personal access token

  • While on Github, click your profile icon > Settings
  • Then click Developer settings > Personal access tokens
  • Click Generate new token
  • On the next page, give the token a name so you know what its for
  • Scroll down to the bottom and click Generate token
  • Copy the token string and paste it into the GITHUB_ACCESS_TOKEN field in the .env file. Note that you will not be able to see this token string on GitHub again

4. Create your own GitHub app

  • While on GitHub, click your profile icon > Settings
  • Then click developer settings > New GitHub app
  • Fill in the name field with a name of your choice
  • Fill in the Homepage URL field with the URL to the GitHub repository for your app. e.g. https://github.com/username/contribute
  • In a new tab, go to https://smee.io/
  • Click Start a new channel and copy the URL they give you. You can ignore the rest of the instructions on the smee.io page
  • Paste the URL you copied into the WEBHOOK_PROXY_URL field of the .env file and the Webhook URL field of your new GitHub app
  • Fill in the Webhook secret field on the GitHub app with a secret of your choice
  • Put the same secret you just used in the WEBHOOK_SECRET field of the .env file
  • Scroll to the bottom of your GitHub app page and click Create GitHub App
  • On the next page, copy the App ID and paste it into the APP_ID field of the .env file
  • Scroll to the bottom of this page and click Generate Private Key
  • A popup menu will allow you to download the key file. Download it and put it in the probot folder

5. Run mongoDB

  • Make sure a mongoDB instance is running by running the command below in the terminal.
mongod —dbpath=./database_folder

6. Start the program

  • In a new terminal window or tab, run these three commands to start the program. Wait for one command to finish running before starting the next.
npm install
npm run build
npm run develop
  • If you skipped steps 2-5, you can ignore the rest

7. Update the Database

  • Run the command below to populate your local database with PRs from the freeCodeCamp repo. Note that you must have the program running and mongoDB running.
node probot/server/tools/update-db.js
  • This will take a while. If it stops running partway through, it's probably a timeout error. Run the command again and it should finish

Caveats & Notes

Local Ports when developing locally

Using npm run develop will start both the api server and the Create React App(Dashboard) in development mode. The api server runs on port 3001 and the React app runs on port 3000.

The one-off scripts will error out on actions performed by repository admins

For example, if an admin removes a label from a Pull Request, the script can not add that label back. This is usually because the script is acting on behalf of a non-admin user with write access. This is usually the case with the use of access tokens for scripts.

Setting up Cron jobs for Sweeper Scripts

For updating dashboard data we use PM2 like so:

pm2 start --no-autorestart probot/server/tools/update-db.js --cron "*/10 * * * *"

This will start the script in the "no restart" mode and re-run it every 10 minutes. A useful link to calculate a Cron expression: https://crontab.guru/every-10-minutes

Starting the express server (via probot)

pm2 start "npm start" --name "contribute-app"

Note: Start only one instance of this app, you can't have multiple probot apps running. Starting multiple instances will crash the app.