Python3 endless fix

We now prominently show that Python2 and Python3 are both supported. WJT
and Adar have been added to contributors too.

Without this, running this code on python2 fails with the following:
`UnicodeDecodeError: 'ascii' codec can't decode byte ... in position ...: ordinal not in range(128)`

Python3 is fine since it uses utf8 by default.

Update syntax to Python3.

Adar Nimrod authored Mar 05, 2020 and nimrod committed Mar 05, 2020

Allow for running under Python 2.6 or 2.7. The invocation needs to be
`python update_repos`.

Adar Nimrod authored Mar 05, 2020 and nimrod committed Mar 05, 2020

Responses from urlopen are bytes. Decode them using the default locale
encoding. Supported since Python 2.3.

Adar Nimrod authored Mar 05, 2020 and nimrod committed Mar 05, 2020

When passing universal_newlines=True to Popen stdout and stderr are text
streams instead of bytes. Supported since Python 3.2.

Adar Nimrod authored Mar 04, 2020 and nimrod committed Mar 04, 2020

Used 2to3. Update shebang to python3 as well.

Sync our fork with upstream now that our fixes have been pushed there. I
opted to merge rather then rebase since the recent revert commit
messages are useful.

`###Some text` -> `### Some text`

Fixes for upstream

GitHub API authentication using the `access_token` query paramater has
been deprecated[1]. Instead, use HTTP basic authentication. This
requires using a `urllib2.Request` object to supply additional headers.
In the future this code should probably switch to using `requests`.

1. https://developer.github.com/changes/2019-11-05-deprecated-passwords-and-authorizations-api/#authenticating-using-query-parameters

https://phabricator.endlessm.com/T29306

Including the `?` and `&` in the templates is awkward. Instead, use a
dictionary with just the parameter names and use `urlencode` to
construct the query string. This also has the advantage that any values
will be properly HTML encoded.

https://phabricator.endlessm.com/T29306

git help fetch says:

> By default, any tag that points into the histories being fetched is
> also fetched; the effect is to fetch tags that point at branches that
> you are interested in.

To give an example:

    $ mkdir /tmp/source
    $ cd /tmp/source
    $ git init
    $ touch /tmp/source/foo && git add foo && git commit -am foo
    $ git clone /tmp/source /tmp/dest

    $ git tag v1.0
    $ git checkout -b temp
    $ echo "Don't lose this" > foo && git commit -am "Important"
    $ git tag dont-lose-this
    $ git checkout master
    $ git branch -D temp

    $ cd /tmp/dest
    $ git fetch
    From /tmp/source
     * [new tag]         v1.0       -> v1.0
    $ git fetch --tags
    From /tmp/source
     * [new tag]         dont-lose-this -> dont-lose-this

It's important that we fetch all tags because we sometimes tag a branch
with 'archive/something' before force-pushing to it so we don't lose the
old head of that branch; it would be sad to not back those up.

Fix API authentication

GitHub API authentication using the `access_token` query paramater has
been deprecated[1]. Instead, use HTTP basic authentication. This
requires using a `urllib2.Request` object to supply additional headers.
In the future this code should probably switch to using `requests`.

1. https://developer.github.com/changes/2019-11-05-deprecated-passwords-and-authorizations-api/#authenticating-using-query-parameters

https://phabricator.endlessm.com/T29306

Including the `?` and `&` in the templates is awkward. Instead, use a
dictionary with just the parameter names and use `urlencode` to
construct the query string. This also has the advantage that any values
will be properly HTML encoded.

https://phabricator.endlessm.com/T29306

Revert "Prune while fetching"

This reverts commit 17675937.  At
https://github.com/endlessm/gh_fetch/pull/3#issuecomment-507526552
@dbnicholson said:

> This defeats the purpose of the backup tool. The backup should keep
> the refs in case the remote has accidentally deleted them. For instance,
> assume that the eos3.4 branch was deleted on the remote. After this
> change, the backup will follow and we'll no longer have any reference to
> the referenced commit.
>
> In 5 years of running this code, this is the 2nd time we've ever run
> into this kind of branch conflict. I don't think we should weaken the
> backup tool to cater to it.

https://phabricator.endlessm.com/T27103

Revert "Prune tags as well"

This reverts commit 09aa7c81.

Git on the relevant Jenkins job apparently predates 2.17.0 which was
released in June 2018.
https://github.com/git/git/blob/master/Documentation/RelNotes/2.17.0.txt#L32-L33

    - master @ 843dcde ..Could not execute git fetch --prune --tags --prune-tags
    error: unknown option `prune-tags'

https://phabricator.endlessm.com/T27103

Fix backups by pruning old branches

If we delete a tag, we probably have a good reason.

git help fetch says:

> By default, any tag that points into the histories being fetched is
> also fetched; the effect is to fetch tags that point at branches that
> you are interested in.

To give an example:

    $ mkdir /tmp/source
    $ cd /tmp/source
    $ git init
    $ touch /tmp/source/foo && git add foo && git commit -am foo
    $ git clone /tmp/source /tmp/dest

    $ git tag v1.0
    $ git checkout -b temp
    $ echo "Don't lose this" > foo && git commit -am "Important"
    $ git tag dont-lose-this
    $ git checkout master
    $ git branch -D temp

    $ cd /tmp/dest
    $ git fetch
    From /tmp/source
     * [new tag]         v1.0       -> v1.0
    $ git fetch --tags
    From /tmp/source
     * [new tag]         dont-lose-this -> dont-lose-this

It's important that we fetch all tags because we sometimes tag a branch
with 'archive/something' before force-pushing to it so we don't lose the
old head of that branch; it would be sad to not back those up.

This job has been failing for the last week:

    ----------------------eos-openqa-tests----------------------
    Using /opt/backups/eos-openqa-tests.git
    - master @ 0c013e2 ..Could not execute git fetch
    error: cannot lock ref 'refs/heads/staging': 'refs/heads/staging/T25144-fix-fbe_language_switch-on-ISOs' exists; cannot create 'refs/heads/staging'
    From github.com:endlessm/eos-openqa-tests
     ! [new branch]      staging    -> staging  (unable to update local ref)
    Terminating early

This is because we previously had branches of the form 'staging/*' in
this repo, and we now have only a 'staging' branch (or none). Git
doesn't support both at once, but nothing pruned old staging branches.
Prune 'em.

https://phabricator.endlessm.com/T27103

Fixed issues with null descriptions

Fixed issues with null descriptions

API changed to now include nulls in descriptions so we needed to account
for that case in the logic.

fix finding org repos for backup

move organization URL to Constant.

fix finding org repos for backup

fixes#11

The endpoint for organizations we're hitting is only for public membership. Instead use endpoint that returns all organization membership for the authenticated user.

See: https://developer.github.com/v3/orgs/

  Github was maxing the requests per page to 100 regardless to what was
  requested so paging functionality was added to work around it.

Added handling of wikis that have been removed

  Previously the wikis that we have on disk were attempted to be fetched
  regardless of if the wiki was avaliable or not so now we do an explicit
  check to ensure that in the case of a deleted wiki, we do not break the
  whole process.

  Edit: Same fetch check has been added to source repos as well