diff options
| -rw-r--r-- | README.txt | 107 | ||||
| -rw-r--r-- | support/README.txt | 6 | ||||
| -rw-r--r-- | support/schema/aur-schema.sql | 139 | ||||
| -rw-r--r-- | tupkg/README.txt | 99 | ||||
| -rwxr-xr-x | tupkg/client/tupkg | 19 | ||||
| -rwxr-xr-x | tupkg/server/tupkgs | 22 | ||||
| -rw-r--r-- | web/README.txt | 94 |
7 files changed, 486 insertions, 0 deletions
diff --git a/README.txt b/README.txt new file mode 100644 index 0000000..12f64c7 --- /dev/null +++ b/README.txt @@ -0,0 +1,107 @@ +This is the finalized draft of the project requirements for the +new Arch package submittal process. AUR (Arch User-community Repo). +The sub-directories contain more specific implementation notes +for each component of the project. + + +Requirements: +------------- +1) User accounts (users, TUs) + - Create account. (email address required) + - Update account (change password/email address) + - Log in/out + +2) Search for packages (public) + - needs knowledge of ALL pkgs (OfficalRepos/AUR/Unsupported). This + should be easy enough if this site lives on the same machine as + the current package database (dragon?), or is allowed to query + the db. + - Display official repo (current/extra) a package lives in. + +3) Manual voting (requires user acct) + - reset/clear all votes (for starting over, this can be added later + if there is any demand for it) + +4) Package Management + - A package can be submitted by anyone (as long as they create + an account with a valid email address on the web site). From + there, a TU inspects the package and works with the submitter + to resolve any bugs/issues. Once approved, the TU can place the + package in the AUR. From there, if the package is popular enough + (or deemed necessary), a developer can move the package from the + AUR to extra/current/etc. A developer can also downgrade a + package from extra/current/etc to the AUR. + - The person that uploaded the new package can make changes to + it before it has been added to the AUR. + - TUs need to be able to purge packages in "Unsupported" if the + package is no longer maintained and there is no interest in + keeping it. + - Packages in the AUR/Unsupported need some sort of 'flag out of + date' support. + - Interested users can download the package from "Unsupported" + and build/install it by hand. + - Provide a separate installation of flyspray for tracking bugs + for packages living in the AUR. All bugs should be resolved + in either flyspray (AUR/official) prior to a package being + promoted/demoted. + +5) Reports + - package popularity by number of votes + +6) Wiki Docs (UID/GID db, provides db, irc nicks/names TUs/devs) + - Move the appropriate dev wiki pages to the new system's + wiki area. The devs will just need to consult the UID/GID + list from the new system site rather than our own wiki. + +7) Submitting 'new' packages by users. Initially start with + a simple web upload page where users submit a tgz containing + the PKGBUILD, scriptlets, patches, etc. The script will + unpack the tgz in an appropriate location and update the + backend database to 'register' the pacakge. + +8) TU package management + - A TU adopts a package from "Unsupported" and that shows users + and other TUs that the package is being reviewed. + - When the TU is ready to move the package to the AUR, they + use a custom utility/script that allows them to upload the + pkg.tar.gz (web uploads are inadequate for this process). + The upload utility/script has a server counterpart that + performs TU authentication and updates the database. + - A cronjob on the server can handle the new AUR package, + update the database, and rebuild the AUR sync db, and send + email notices to the TU about errors if any occur. + - The TUs should also be able to demote a package from the + AUR via the web interface. + - TUs will use cvs/svn interface (just like devs) to pull + down the PKGBUILD tree. This tree should reflect the same + layout as extra for easier package migration. They make + changes to their local copy, test, and then commit. They + use the xfer utility to upload the binary package to the + server. No shell access is required. + + +Automated Voting Tool (similar to ArchStats client) +===================== + +Requirements: +------------- + 1) Name of tool is 'pkgvote' + + 2) Requires registered account on web - email address not required + + 3) Casts 'yes' votes for all installed packages (except itself?) + +Implementation: +--------------- + A statically compiled C program that gathers the list of installed + packages and casts the vote to the web site. Very similar to the + way that ArchStats works now. When making the HTTP Post, it adds + a custom HEADER that the PHP script(s) can detect to know that it + is receiving a vote from a 'pkgvote' client. If the PHP script + does not see the special HEADER, it assumes it is a web browser + and gives the user HTML output. + + Once installed, the user edits the config file and supplies their + username/password. If no username/password exists in the config + file when it starts, it spits out an error message and exits. + diff --git a/support/README.txt b/support/README.txt new file mode 100644 index 0000000..8b01ccb --- /dev/null +++ b/support/README.txt @@ -0,0 +1,6 @@ +The schema dir contains the AUR database schema file. + +The scripts dir will contain any supporting scripts for the AUR such +as the script that handles moving newly uploaded packages via tupkg +to the AUR. + diff --git a/support/schema/aur-schema.sql b/support/schema/aur-schema.sql new file mode 100644 index 0000000..e147c47 --- /dev/null +++ b/support/schema/aur-schema.sql @@ -0,0 +1,139 @@ +-- The MySQL database layout for the AUR. Certain data +-- is also included such as AccountTypes, PackageLocations, etc. +-- + +-- Define the Account Types for the AUR. +-- +CREATE TABLE AccountTypes ( + ID UNSIGNED TINYINT NOT NULL AUTO_INCREMENT, + AccountType char(32) NOT NULL DEFAULT '', + PRIMARY KEY (ID) +); +INSERT INTO TABLE (ID, AccountType) VALUES (1, 'User'); +INSERT INTO TABLE (ID, AccountType) VALUES (2, 'Trusted User'); +INSERT INTO TABLE (ID, AccountType) VALUES (3, 'Developer'); + + +-- User information for each user regardless of type. +-- +CREATE TABLE Users ( + ID UNSIGNED INTEGER NOT NULL AUTO_INCREMENT, + AccountTypeID UNSIGNED TINYINT NOT NULL DEFAULT 1, + Suspended UNSIGNED TINYINT NOT NULL DEFAULT 0, + Email CHAR(64) NOT NULL, + Passwd CHAR(32) NOT NULL, + RealName CHAR(64) NOT NULL DEFAULT '', + IRCNick CHAR(32) NOT NULL DEFAULT '', + LastVoted UNSIGNED BIGINT NOT NULL DEFAULT 0, + NewPkgNotify UNSIGNED TINYINT NOT NULL DEFAULT 0, + PRIMARY KEY (ID), + UNIQUE INDEX Emailx (Email), + INDEX AccountTypeIDx (AccountTypeID), + INDEX NewPkgNotifyx (NewPkgNotify), + FOREIGN KEY AccountTypeIDr REFERENCES AccountTypes (ID) +); +-- A default developer account for testing purposes +INSERT INTO Users (ID, AccountTypeID, Email, Passwd) VALUES ( + 1, 3, 'root@localhost', 'changeme'); + + +-- Track Users logging in/out of AUR web site. +-- +CREATE TABLE Sessions ( + UsersID UNSIGNED INTEGER NOT NULL, + SessionID CHAR(32) NOT NULL, + LastUpdateTS UNSIGNED BIGINT NOT NULL, + FOREIGN KEY UsersIDr REFERENCES Users (ID) +); + + +-- Categories for grouping packages when they reside in +-- Unsupported or the AUR - based on the categories defined +-- in 'extra'. +-- +CREATE TABLE PackageCategories ( + ID UNSIGNED TINYINT NOT NULL AUTO_INCREMENT, + Category CHAR(32) NOT NULL, + PRIMARY KEY (ID) +); +INSERT INTO PackageCategories (Category) VALUES ('daemons'); +INSERT INTO PackageCategories (Category) VALUES ('devel'); +INSERT INTO PackageCategories (Category) VALUES ('editors'); +INSERT INTO PackageCategories (Category) VALUES ('emulators'); +INSERT INTO PackageCategories (Category) VALUES ('games'); +INSERT INTO PackageCategories (Category) VALUES ('gnome'); +INSERT INTO PackageCategories (Category) VALUES ('i18n'); +INSERT INTO PackageCategories (Category) VALUES ('kde'); +INSERT INTO PackageCategories (Category) VALUES ('lib'); +INSERT INTO PackageCategories (Category) VALUES ('modules'); +INSERT INTO PackageCategories (Category) VALUES ('multimedia'); +INSERT INTO PackageCategories (Category) VALUES ('network'); +INSERT INTO PackageCategories (Category) VALUES ('office'); +INSERT INTO PackageCategories (Category) VALUES ('science'); +INSERT INTO PackageCategories (Category) VALUES ('system'); +INSERT INTO PackageCategories (Category) VALUES ('x11'); +INSERT INTO PackageCategories (Category) VALUES ('xfce'); + + +-- The various repositories that a package could live in. +-- +CREATE TABLE PackageLocations ( + ID UNSIGNED TINYINT NOT NULL AUTO_INCREMENT, + Location CHAR(32) NOT NULL, + PRIMARY KEY (ID) +); +INSERT INTO PackageLocations (ID, Location) VALUES (1, 'Unsupported'); +INSERT INTO PackageLocations (ID, Location) VALUES (2, 'AUR'); +INSERT INTO PackageLocations (ID, Location) VALUES (3, 'Current'); +INSERT INTO PackageLocations (ID, Location) VALUES (4, 'Extra'); +INSERT INTO PackageLocations (ID, Location) VALUES (5, 'Unstable'); + + +-- Information about the actual packages +-- +CREATE TABLE Packages ( + ID UNSIGNED INTEGER NOT NULL AUTO_INCREMENT, + Name CHAR(32) NOT NULL, + Version CHAR(32) NOT NULL DEFAULT '', + CategoryID UNSIGNED TINYINT NOT NULL, + Description CHAR(128) NOT NULL DEFAULT "An Arch Package", + URL CHAR(256) NOT NULL DEFAULT "http://www.archlinux.org", + Source CHAR(256) NOT NULL DEFAULT "/dev/null", + LocationID UNSIGNED TINYINT NOT NULL, + OutOfDate UNSIGNED TINYINT DEFAULT 0, + SubmittedTS UNSIGNED BIGINT NOT NULL, + SubmitterUID UNSIGNED INTEGER NOT NULL DEFAULT 0, + MaintainerUID UNSIGNED INTEGER NOT NULL DEFAULT 0, + PRIMARY KEY (ID), + UNIQUE INDEX Namex (Name), + INDEX CategoryIDx (CategoryID), + INDEX LocationIDx (LocationID), + INDEX OutOfDatex (OutOfDate), + INDEX SubmitterUIDx (SubmitterUID), + INDEX MaintainerUIDx (MaintainerUID), + FOREIGN KEY CategoryIDr REFERENCES PackageCategories (ID), + FOREIGN KEY LocationIDr REFERENCES PackageLocations (ID) + FOREIGN KEY SubmitterUIDr REFERENCES Users (ID) + FOREIGN KEY MaintainerUIDr REFERENCES Users (ID) +); + + +-- Track votes for packages +-- +CREATE TABLE PackageVotes ( + UsersID UNSIGNED INTEGER NOT NULL, + PackageID UNSIGNED INTEGER NOT NULL, + PRIMARY KEY (ID), + FOREIGN KEY UsersIDx REFERENCES Users (ID), + FOREIGN KEY PackageIDx REFERENCES Packages (ID) +); + + +-- The individual files and their file system location. +-- +CREATE TABLE PackageContents ( + PackageID UNSIGNED INTEGER NOT NULL, + Path CHAR(256) NOT NULL, + INDEX PackageIDx (PackageID) +); + diff --git a/tupkg/README.txt b/tupkg/README.txt new file mode 100644 index 0000000..47facc4 --- /dev/null +++ b/tupkg/README.txt @@ -0,0 +1,99 @@ +TU Packaging Tools (tupkg) +-------------------------- +- client side (python for proof of concept, later re-write to C?) + The main purpose of this tool is to upload the compiled + pkg.tar.gz to the server. It can (should?) do some verification + on the package prior to uploading to the server. It will have + a config file to store run-time information such as username + (email), password, and server name. + +- server side (python for proof of concept, later re-write to C?) + The server side will handle incoming connections from its client + side counterpart. The server should bind to port 80 (maybe a + vhost such as tupkg.archlinux.org?) so that firewalls won't be + an issue. The server verifies the client authentication data, + and then accepts the package(s). If port 80 is not available, + perhaps 443, or are there other 'standard' ports that usually + do not get filtered? + + I think the server should be multithreaded to handle simultaneous + uploads rather than queue up requests. The download should be + stored in a temp directory based on the username to prevent + directory, filename clashes. + + Once the package(s) is uploaded, the server can either kick off + a gensync, or we can write a separate script to call gensync once + or twice a day. My preference would be a separate script to call + gensync (like the *NIX philosophy of one tool per task). + +- protocol (c: => client, s: => server) + Whenever the client/server exchange a message, it is always + preceeded by two-bytes representing the following message's + length. For example, when the client connects, it will send: + + 0x0028username=bfinch@example.net&password=B0b + + 0x0028 is the 40 byte strlen of the message in two-bytes. The + client and server always read two-bytes from the socket, and + then know how much data is coming and can read that amount of + bytes from the socket. + + ==> authentication + c: username=emailaddy&password=mypassword + s: result=PASS|FAIL + + NOTE: We can add encryption easily enough with the python + version using the socket.ssl method. + + ==> uploading package data + if PASS: + + c: numpkgs=2&name1=p1.pkg.tar.gz&size1=123&md5sum1=abcd\ + name2=p2.pkg.tar.gz&size2=3&md5sum2=def1 + s: numpkgs=2&name1=p1.pkg.tar.gz&size1=119&\ + name2=p2.pkg.tar.gz&size2=0 (*) + + (*) NOTE: The server will reply back to the client how many + packages it has already received and its local file size. + This way, the client can resume an upload. In the example + above, the server still needs the last four (123-119) bytes + for the first package, and that it has no part of the + second package. The client would then begin sending the + last four bytes from the first package (p1.pkg.tar.gz) and + then follow it with the full second package (p2.pkg.tar.gz). + The data would be sent as a continuous chunk of data. The + server will then need to track which bytes belong to which + package. + + else FAIL: + c: -spits out error message on stderr to user- + + + ==> after upload completes + The server should verify the integrity of the uploaded packages + by doing an md5sum on each and sending the info back to the client + for comparison. After sending the message, the server waits for + the 'ack' message from the client and then closes the connection. + + s: np=2&m1=PASS&m2=FAIL + c: ack + + The client replies with the 'ack' and then closes its connection + to the server. It then reports the PASS/FAIL status of each + package's upload attempt. + + NOTE: If the upload fails (client connection dies), the server + keeps any data it has received in order to support resuming an + upload. However, if the client uploads all data, and the server + successully reads all data and the final MD5 fails, the server + deletes the failed package. + + +Terms/definitions: +====================== +TU - No change (trusted by the community, if anyone asks what trust + means) +TUR - renamed to Arch User-community Repo (AUR) (so we can use -u for + versions) +Incoming - renamed to "Unsupported" + diff --git a/tupkg/client/tupkg b/tupkg/client/tupkg new file mode 100755 index 0000000..ab42b43 --- /dev/null +++ b/tupkg/client/tupkg @@ -0,0 +1,19 @@ +#!/usr/bin/python -O +# +# Description: +# ------------ +# This is the client-side portion of the Trusted User package +# manager. The TUs will use this program to upload packages into +# the AUR. For more information, see the ../README.txt file. +# +# Python Indentation: +# ------------------- +# Use tabs not spaces. If you use vim, the following comment will +# configure it to use tabs. +# vim: ts=2 sw=2 noet ft=python +# + + + +# TODO write the code +# diff --git a/tupkg/server/tupkgs b/tupkg/server/tupkgs new file mode 100755 index 0000000..2c410a5 --- /dev/null +++ b/tupkg/server/tupkgs @@ -0,0 +1,22 @@ +#!/usr/bin/python -O +# +# Description: +# ------------ +# This is the server-side portion of the Trusted User package +# manager. This program will receive uploads from its client-side +# couterpart, tupkg. Once a package is received and verified, it +# is placed in a specified temporary incoming directory where +# a separate script will handle migrating it to the AUR. For +# more information, see the ../README.txt file. +# +# Python Indentation: +# ------------------- +# Use tabs not spaces. If you use vim, the following comment will +# configure it to use tabs. +# vim: ts=2 sw=2 noet ft=python +# + + + +# TODO write the code +# diff --git a/web/README.txt b/web/README.txt new file mode 100644 index 0000000..5795705 --- /dev/null +++ b/web/README.txt @@ -0,0 +1,94 @@ +Web Interface: +============== + +Directory Layout: +----------------- +./html - DocumentRoot for AUR, where the PHP scripts live. +./html/css - CSS stylesheets +./html/images - Any AUR images live here. +./lib - Supporting PHP include files. Access denied to Apache. + + +Scripts: +-------- +- lib/funcs.inc + This is where we can stick functions that can be shared + between the various scripts. Also a good place to put the + MySQL authentication variables since it should live outside + the DocumentRoot. + +- html/login.php (probably index.php) + PHP script to handle logging users into the AUR web site. It + authenticates using the email address and a password against + the Users table. Once authenticated, a session id is generated + and stored in the Sessions table and sent as a cookie to the + user's browser. + +- html/logout.php + PHP script to logout. It clears the session id from the + Sessions table and unsets the cookie. + +- html/account.php + PHP script to handle registering for a new account. It prompts + the visitor for account information: Email, password, real name, + irc nick. The info is recorded in the Users table. Perhaps later, + we can add a preferences field that allows the user to request to + be notified when new packages are submitted so that they can cast + votes for them? + + If a TU is logged into the system, they can edit accounts and set + the account type (regular user or TU). If a Dev is logged in, they + can also set the account type to Dev. TUs and Devs are able to + delete accounts. If an account is deleted, all "Unsupported" + packages are orphaned (the Users.ID field in the UnsupportedPackages + table is set to Null). + +- html/pkgsearch.php + PHP script to search the package database. It should support + searching by location ("unsupported", "AUR", "extra"), name, + category, maintainer, popularity, etc. It should resemble the + packages.php script on archlinux.org. A checkbox should be + included next to each package to allow users to flag a package + out of date. + +- html/pkgvote.php + The PHP script that handles voting for packages. It works like + the search script above to provide a list of packages (maybe by + location only?) with a checkbox for the user to register their + 'yes' vote. It should probably only list 50-ish packages per page + and allow the user to vote a page at a time. Each page contains a + 'Continue' button to advance to the next list of packages. At the + final page, a summary is presented with a 'Cast Vote' button. Once + the vote is cast, the PackageVotes table is first cleared for that + User and then all new entries are added for the new vote (this will + be easier than trying to figure out if the vote has changed for a + particular package). + +- html/pkgmgmnt.php + The PHP script for managing packages. It provides a list of + packages under the management of the logged in user (normal or + TU). The user can edit the package information stored in the + database such as the description, url, category, and location + (a TU can move it to/from "unsupported" and the "AUR"). This + is where TUs can adopt packages that are in the "unsupported" + area. + +- html/pkgsubmit.php + This is the PHP script that allows users to upload a new package. + The package format will be a tgz containing the PKGBUILD, + scriptlets, and patches necessary to build the package from + source. Initially, the user submitting the package can select + its category (network, devel, etc) but that can be modified + later by the adopting TU. The script makes appropriate entries + into the database (and perhaps notifies interested users of the + new package - see question above). + + +New terms/definitions: +====================== +TU - No change (trusted by the community, if anyone asks what trust + means) +TUR - renamed to Arch User-community Repo (AUR) (so we can use -u for + versions) +Incoming - renamed to "Unsupported" + |