diff --git a/.gitignore b/.gitignore index d591df4c25a4dd2c78a1c40a13628e800e7e067a..6d3bf782c30ceae9c68c6f08f817fe14366b1cae 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,4 @@ repos jails logs +PACKAGES/ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000000000000000000000000000000000000..b10a25f4a9f57b46bb19f64f262b81543813e6fb --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,112 @@ +# Contributing + +Whether you've got a bugfix, documentation update, or new feature for us, these are the steps to follow to contribute code back into the main trisquel repo. + +## 1. Prerequisites + + + + 1. Install needed packages for both, sources and binary builds +``` +sudo apt-get install dpkg-dev devscripts git pbuilder quilt patch sed rpl parsewiki +``` + + 1. Configure build environment for binary packages +``` +git clone https://devel.trisquel.info/trisquel/trisquel-builder.git +ln -s $(readlink -f trisquel-builder/pbuilderrc) ~/.pbuilderrc +sudo ln -s $(readlink -f trisquel-builder/hooks) /var/cache/pbuilder/hooks.d +``` + + 1. Create the build environment for each distribution/architecture you want to work with +``` +sudo BUILDDIST=belenos BUILDARCH=amd64 pbuilder create +``` + + +## 2. Get the latest code from gitlab + +You'll need to understand a little bit about how git and gitlab work before this step (GitHub works the same way, but we like free software). In simple terms, log in [here](https://devel.trisquel.info/users/sign_in?redirect_to_referer=yes), visit the [trisquel/package-helpers project page](https://devel.trisquel.info/trisquel/package-helpers), and click the "fork" button to create your own copy of the repo. You will push your changes to this new repo under your own git account, and we will pull changes into the main repo from there. + +For the sake of the rest of the examples in this guide, we're going to assume your gitlab username is "**richardtorvalds**" and you will be working with the "**hello**" package, and use those in our examples. + +Now, we want to grab the latest from this newly created repository and pull it down to your local machine. Getting the latest code from your repo is simple, just clone it: + +```bash +git clone https://devel.trisquel.info/richardtorvalds/package-helpers.git +cd package-helpers +``` + +This will give you a directory called "package-helpers" on your local machine with the latest checkout from your fork of the main package-helpers repo. **Note:** this is *not* a direct reference to the main trisquel repo. When you make changes in your fork, you'll need to let us know about it so we can pull it over....but that's later in the process. + +## 3. Add a remote for the main package-helpers repo + +One thing you'll need to do to make things easier to integrate and keep up to date in your fork is to add the main repo as a remote reference. This way you can fetch the latest code from the production version and integrate it. So, here's how to set that up: + +```bash +git remote add upstream https://devel.trisquel.info/trisquel/package-helpers.git +git remote (this will list out your remotes, showing the new one we added) +``` + +Then, when you need to pull the latest from the main trisquel repo, you just fetch and merge the master branch: + +```bash +git fetch upstream +git merge upstream/belenos +``` + +You can also use *git pull upstream belenos* if you want it all in one step. + +## 4. Making a branch for your changes + +When adding features or bug fixes, please create a separate branch for each changeset you want us to pull in, either with the issue number in the branch name or with an indication of what the feature is (feature, bugfix...). + +```bash +git branch (lists your current branches) +git checkout -b bugfix-hello (makes a new branch called bugfix-hello) +``` + + +## 5. Building the source package + +If you want to create a new package, apache2 is a good starting point, so: +``` +cd helpers +cp make-apache2 make-hello +``` + +For importing free packages from ppa's or other sources, check [make-toxcore](https://devel.trisquel.info/trisquel/package-helpers/blob/belenos/helpers/make-toxcore) and update the _EXTERNAL_ , _SIGNKEY_ and _changelog_ lines with your own values. + +Then, run the helper with +``` +bash make-hello +``` + +If everything goes fine, you will have your new source package ready at _PACKAGES/hello/_ + +## 6. Build and test the binary package + +The last step generated a source package file, so we need to build the binary one: +``` +sudo BUILDDIST=belenos BUILDARCH=amd64 pbuilder build PACKAGES/hello/*.dsc +``` + +The binary packages will be avaliable at _/var/cache/pbuilder/jenkins-repos/$BUILDDIST/_ + + +## 7. Push your code and make a pull request + +When you have finished making your changes, you'll need to push up your changes to your fork so we can grab them. With them all committed, push them: + +```bash +git push origin bugfix-hello +``` + +This pushes everything in that branch up. Then you can go back to your forked package-helpers gitlab page and issue a pull request from there. Tell us what you want us to merge and what it does/fixes, and one of us will pick it up. + +That lets us know that there's something new from you that needs to be pulled in. We'll review it and get back to you about it if we have any questions. Otherwise, we'll integrate it and let you know when it's in! + + +Hope this guide helps you get started in contributing to the trisquel project! If you still have questions, don't hesitate to join us on IRC - we're in #trisquel-dev on freenode -, or send a mail to the development mailing list trisquel-devel at listas.trisquel.info. + + diff --git a/README.md b/README.md index d61aba61e0fbeaa05d5af71592bbcfbe79c87923..aa86fa6576fc94c776d47c3e661ee920906669c9 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ ## Introduction -This set of scripts are helpers that modify and compile those packages coming +This set of scripts are helpers that modify and create the source packages coming from the Ubuntu upstream which need it. It might be because they contain non-free stuff, references to Ubuntu that need to be changed, or because we want the package to work our way. @@ -9,20 +9,29 @@ This helpers are similar to some of those in the [gNewSense](http://www.gnewsens some ideas and even some lines from them. If you plan to build an Ubuntu derivative of your own, we suggest you to use Builder instead of this helpers. -All packages in this list are in the appropriate blacklist in the repository -updater, so they never enter into the repo from upstream and need to be -compiled with this helpers and pushed into reprepro. This helpers also need -to be run by hand - and the results tested - any time the repo watchdog warns -about pending updates from upstream. +Once a new package is added, it takes priority over the original one from Ubuntu, +so they never enter into the repo from upstream and need to be +compiled with this helpers and pushed into reprepro. -To add a package to the list, just copy one - `make-apache2` is a good template - -and rename it to `make-sourcePackageName`. To send the file back to us, or to -include any modification into the current scripts, use `bzr diff` and send -the output as an attachment to trisquel-devel@listas.trisquel.info. You need -to join the mailing list to send messages to it. +To add a package to the list, follow the [CONTRIBUTING](https://devel.trisquel.info/trisquel/package-helpers/blob/belenos/CONTRIBUTING.md) guidelines. + +## Steps + +Those are the steps done by the helpers: + +1. Create local apt configuration, so you don't need to be root to run the helpers +2. Get the ubuntu and trisquel gpg keys +3. Get the source packages from their original repo +4. Uncompress them +5. Apply the changes described in the helper +6. Re-package it, adding "triquel$VERSION" version string ## Recommendations +* You don't need to use sudo in order to run those scripts, but some extra packages are needed: + + `sudo apt-get install dpkg-dev sed git rpl devscripts quilt patch` + * Take care to use the right sourcePackageName, many source packages produce several binary packages. `apt-cache showsrc binary-package` can help you. * If possible, use sed to replace chains in the upstream source without the @@ -32,9 +41,8 @@ it at the `DATA/sourcePackageName` directory would actually be shown to the user. Avoid replacing copyright statements! * Try to write your replacements in a way they might work in future versions of the upstream package. Well written regexps and sed will help with that. -* Some of this packages require the lsb to match Trisquel values. Edit the -`/etc/lsb_release` accordingly, or run the helpers in a Trisquel box. -* You can - and maybe should - run this scripts inside a chroot. +* You can test your changes by doing them inside the _PACKAGES/sourcePackageName/source/_ directory, +and running `dpkg-source -b .`, before being added to the helper script ## Netinstall @@ -54,3 +62,4 @@ To generate the images, we run the following scripts: Then we push the results into the Trisquel repository and run the script `make-debian-installer` to build the final images. + diff --git a/makepackage b/makepackage deleted file mode 100644 index 5063febf9aa117f57142ffb97b3efafcd62b2004..0000000000000000000000000000000000000000 --- a/makepackage +++ /dev/null @@ -1,278 +0,0 @@ -#!/bin/bash -# -# Copyright (C) 2011 Ruben Rodriguez <ruben@trisquel.info> -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA -# - -set -e - -export LD_PRELOAD="${LD_PRELOAD:+$LD_PRELOAD:}/usr/lib/libeatmydata/libeatmydata.so" -export DEBIAN_FRONTEND=noninteractive -MEM=$(cat /proc/meminfo |grep MemTotal| awk '{print $2}') -GPGKEY=8D8AEBF1 - -for VAR in LANG LANGUAGE LC_ALL LC_TIME LC_MONETARY LC_ADDRESS LC_TELEPHONE LC_MESSAGES LC_NAME LC_MEASUREMENT LC_IDENTIFICATION LC_IDENTIFICATION LC_NUMERIC LC_PAPER LANG -do - unset $VAR -done -export LANG=C -export HOME=/root -export DATE=$(date +'%a, %d %b %Y %T %z') - -cd $(dirname $0) - -if [ $# -lt 1 ] -then - echo "Usage: $0 package" - echo example: $0 linux lucid - exit 1 -fi - -PACKAGE=$1 -CODENAME=$(grep 'export CODENAME=' helpers/config |sed 's/.*=//') -UPSTREAM=$(grep 'export UPSTREAM=' helpers/config |sed 's/.*=//') -WORKDIR=tmp/makepackage -REPOSITORY=$PWD/repos/$CODENAME -HELPERS=$PWD/helpers -for REPO in $(git branch|sed 's/.* //;/master/d') -do -[ -d $REPOSITORY ] && continue -mkdir -p repos/$REPO/conf repos/$REPO/incoming -cat << EOF > repos/$REPO/conf/distributions -Origin: Trisquel -Label: Trisquel -Suite: $REPO -Version: 42 -Codename: $REPO -Architectures: i386 amd64 source -Components: main -UDebComponents: main -DebIndices: Packages Release . .gz .bz2 -UDebIndices: Packages . .gz .bz2 -DscIndices: Sources Release .gz .bz2 -Log: $REPO.log -Description: Trisquel GNU/Linux packages for the $UPSTREAM based release -EOF - -if gpg -K | grep -q $GPGKEY -then - echo "SignWith: $GPGKEY" >> repos/$REPO/conf/distributions -fi - -cd repos/$REPO -reprepro -v export -cd ../.. -done - -disablescripts(){ - # Disable service starter scripts - for i in /usr/sbin/invoke-rc.d /sbin/start /sbin/start-stop-daemon - do - mv $CHROOT/$i $CHROOT/$i.real - cp $CHROOT/bin/true $CHROOT/$i - done -} - -createjail () { - [ -d jails ] || mkdir jails - CHROOT=jails/$UPSTREAM-$JAILARCH - C="chroot $CHROOT" - MIRROR=http://mir1.ovh.net/mirrors/ftp.ubuntu.com/ubuntu - - mkdir $CHROOT-tmp - mount -t tmpfs -o size=500M none $CHROOT-tmp - debootstrap --arch=$JAILARCH $UPSTREAM $CHROOT-tmp $MIRROR - fuser -k $CHROOT-tmp - cp -a $CHROOT-tmp $CHROOT - umount $CHROOT-tmp - rm $CHROOT-tmp -rf - echo "127.0.0.1 localhost" > $CHROOT/etc/hosts - mount none $CHROOT/proc -t proc - mount none $CHROOT/dev/pts -t devpts - - disablescripts - - if gpg -K | grep $GPGKEY -q - then - cp /root/.gnupg $CHROOT/root -a - else - echo GPG private key for $GPGKEY not found, the packages and repositories will not be signed. - fi - - echo $UPSTREAM-$JAILARCH > $CHROOT/etc/debian_chroot -cat << EOF > $CHROOT/etc/apt/sources.list -deb $MIRROR $UPSTREAM main universe -deb $MIRROR $UPSTREAM-updates main universe -deb $MIRROR $UPSTREAM-security main universe -deb $MIRROR $UPSTREAM-backports main universe -deb-src $MIRROR $UPSTREAM main universe -deb-src $MIRROR $UPSTREAM-updates main universe -deb-src $MIRROR $UPSTREAM-security main universe -deb-src $MIRROR $UPSTREAM-backports main universe -EOF - -cat << EOF > $CHROOT/etc/apt/apt.conf.d/90recommends -APT::Install-Recommends "0"; -APT::Install-Suggests "0"; -EOF - - export DEBIAN_FRONTEND=noninteractive - - [ $UPSTREAM != hardy -a $UPSTREAM != lucid -a $UPSTREAM != maverick ] && echo "force-unsafe-io" > $CHROOT/etc/dpkg/dpkg.cfg.d/02apt-speedup - $C apt-get update - [ $UPSTREAM != hardy ] && $C apt-get --force-yes -y install sysv-rc dpkg sysvinit-utils upstart - disablescripts - $C apt-get --force-yes -y dist-upgrade - $C apt-get --force-yes -y install devscripts build-essential liburi-perl python-setuptools pkgbinarymangler wget rpl aptitude quilt fakeroot ccache - $C apt-get clean - - wget -o /dev/null -O $CHROOT/tmp/key.gpg http://archive.trisquel.info/trisquel/trisquel-archive-signkey.gpg - $C apt-key add /tmp/key.gpg - - # Hack for i386 - if [ $JAILARCH = "i386" ] - then - for BIN in /bin/uname /usr/bin/arch - do - [ -f $CHROOT/$BIN ] || continue - mv $CHROOT/$BIN $CHROOT/$BIN.orig -cat << EOF > $CHROOT/$BIN -#!/bin/bash - -if [ \$# -eq 0 ] -then - $BIN.orig | sed s/x86_64/i686/g -else - $BIN.orig "\$*" | sed s/x86_64/i686/g -fi -EOF - chmod 755 $CHROOT/$BIN -done - fi -umount $CHROOT/proc $CHROOT/dev/pts -} - -prepare(){ -[ -d jails/$UPSTREAM-$JAILARCH ] || createjail -cat << EOF > jails/$UPSTREAM-$JAILARCH/tmp/update -mount -t proc none /proc -export DEBIAN_FRONTEND=noninteractive -export LANG=C -apt-get update -apt-get --force-yes -y install sysv-rc dpkg sysvinit-utils upstart -for i in /usr/sbin/invoke-rc.d /sbin/start /sbin/start-stop-daemon -do - mv \$i \$i.real - cp /bin/true \$i -done -export DEBIAN_FRONTEND=noninteractive -apt-get --force-yes -y dist-upgrade -umount /proc -EOF -chroot jails/$UPSTREAM-$JAILARCH bash /tmp/update -if [ -f jails/$UPSTREAM-$JAILARCH/CurrentlyBuilding ] -then - echo The $UPSTREAM-$JAILARCH jail appears to be running $(cat jails/$UPSTREAM-$JAILARCH/CurrentlyBuilding |grep Package: |sed 's/Package:\ //'), aborting. - exit 1 -fi - CHROOT=jails/$UPSTREAM-$JAILARCH disablescripts -} - -tmpumount(){ - grep jails/$UPSTREAM-$JAILARCH/tmp /proc/mounts -q || return 0 - umount jails/$UPSTREAM-$JAILARCH/tmp && return - echo ERROR: could not umount tmpfs at jails/$UPSTREAM-$JAILARCH/tmp - exit 1 -} -tmpmount(){ - [ $MEM -lt 16000000 ] && return - grep jails/$UPSTREAM-$JAILARCH/tmp /proc/mounts -q && tmpumount - echo $PACKAGE | grep -q "linux" && return - mount -t tmpfs -o size=20G none jails/$UPSTREAM-$JAILARCH/tmp -} - -compile(){ - [ -d jails/$UPSTREAM-$JAILARCH/$WORKDIR ] && rm -rf jails/$UPSTREAM-$JAILARCH/$WORKDIR/ -tmpmount - cp -a helpers/ jails/$UPSTREAM-$JAILARCH/$WORKDIR/ - -cat << EOF > jails/$UPSTREAM-$JAILARCH/usr/local/sbin/makepackage-handler -#!/bin/bash -WORKDIR=$WORKDIR -PACKAGE=$PACKAGE -cd $WORKDIR -[ -d LOGS ] || mkdir LOGS -[ -d PACKAGES ] || mkdir PACKAGES -export PATH="/usr/lib/ccache:${PATH}" -bash -e make-$PACKAGE 2>&1 || exit 1 -EOF - -cat << EOF > jails/$UPSTREAM-$JAILARCH/etc/pkgbinarymangler/striptranslations.conf -enable: true -components: main -invalid_currentlybuilding: ignore -posuffix: translations -oem_blacklist: partner -EOF - -cat << EOF > jails/$UPSTREAM-$JAILARCH/etc/pkgbinarymangler/sanitychecks.conf -enable: true -EOF - -cat << EOF > jails/$UPSTREAM-$JAILARCH/etc/pkgbinarymangler/maintainermangler.conf -enable: true -invalid_currentlybuilding: ignore -EOF - -cat << EOF > jails/$UPSTREAM-$JAILARCH/etc/pkgbinarymangler/maintainermangler.overrides -default: Trisquel GNU/Linux developers <trisquel-devel@listas.trisquel.info> -ignore_domains: trisquel.info sognus.com listas.trisquel.info gnu.org fsf.org -ignore_emails: ruben@trisquel.info -EOF - - chmod 755 jails/$UPSTREAM-$JAILARCH/usr/local/sbin/makepackage-handler - - if ! chroot jails/$UPSTREAM-$JAILARCH makepackage-handler /$WORKDIR/make-$PACKAGE - then - rm jails/$UPSTREAM-$JAILARCH/CurrentlyBuilding - exit 1 - fi - find jails/$UPSTREAM-$JAILARCH/$WORKDIR/PACKAGES/$PACKAGE/ -maxdepth 1 -type f -exec cp {} $REPOSITORY/incoming/ \; -tmpumount -} - - JAILARCH=i386 prepare - JAILARCH=amd64 prepare - JAILARCH=i386 compile - JAILARCH=amd64 compile - -cd $REPOSITORY - -for dsc in incoming/*.dsc -do - if gpg -K | grep $GPGKEY -q - then - debsign -k$GPGKEY $dsc - else - echo GPG private key for $GPGKEY not found, the packages and repositories will not be signed. - fi - reprepro -v -C main includedsc $CODENAME $dsc || reprepro -S admin -P optional -v -b . -C main includedsc $CODENAME $dsc -done - -find incoming -name *.deb -exec reprepro -v -C main includedeb $CODENAME {} \; -find incoming -name *.udeb -exec reprepro -v -C main includeudeb $CODENAME {} \; - -rm incoming/* diff --git a/makepending b/makepending deleted file mode 100644 index 4547c0eacbfd163882347e0f96430b2ac4443cd5..0000000000000000000000000000000000000000 --- a/makepending +++ /dev/null @@ -1,155 +0,0 @@ -#!/bin/bash -# -# Copyright (C) 2009,2010,2011 Ruben Rodriguez <ruben@trisquel.info> -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA -# -# -# -# This script compares two Sources.gz files, and reports available updates - -if [ $(git status -s | wc -l) != 0 ] -then - echo You have uncommited git changes, stopping - [ 1$1 = 1"test" ] || exit 1 -fi - -PAIRS="taranis-lucid toutatis-precise belenos-trusty" -CODENAMES="taranis toutatis belenos" -UPSTREAMS="lucid precise trusty" - -[ 1$1 = 1"test" ] && TEST=echo - -WD=$PWD -LOGS=$PWD/logs -[ -d $LOGS ] || mkdir $LOGS -export LANG=C -FILE=$(mktemp) -TMP=$(mktemp -d) -cd $TMP - -echo "List of packages with available updates: -" > $FILE - -listmirror (){ -DIST=$1 -COMPONENT=$2 -echo -n Downloading Sources.gz from $MIRROR/dists/$DIST/$COMPONENT/source/Sources.gz >&2 -wget $MIRROR/dists/$DIST/$COMPONENT/source/Sources.gz -q -O - | zcat | egrep dsc$ | cut -d" " -f 4|sed 's/\.dsc//' -echo " Done" >&2 -} - -update(){ - -MIRROR="http://archive.ubuntu.com/ubuntu" - -for i in $UPSTREAMS -do - listmirror $i main >> $i - listmirror $i-updates main >> $i - listmirror $i-security main >> $i - listmirror $i universe >> $i - listmirror $i-updates universe >> $i - listmirror $i-security universe >> $i -done - -for i in $CODENAMES -do - MIRROR="http://archive.trisquel.info/trisquel" - listmirror $i main >> $i - listmirror $i-updates main >> $i - listmirror $i-security main >> $i - - MIRROR="http://devel.trisquel.info/trisquel/$i" - listmirror $i main >> $i -done - -} - -higher(){ -LIST="$(echo $* |sed 's/:/./g; s/+/./g') EOL" - -HI=$(echo $LIST | cut -d" " -f1) -LIST=$(echo $LIST | cut -d" " -f2-) - -while [ "$LIST" != "EOL" ] -do - TEST=$(echo $LIST | cut -d" " -f1) - dpkg --compare-versions $HI lt $TEST 2>/dev/null && HI=$TEST - LIST=$(echo $LIST | cut -d" " -f2-) -done -echo $HI -} - -check(){ -for package in $PACKAGES -do -LIST=$(egrep ^${package}_ $DISTRO) -current=$( higher $LIST) -LIST=$(egrep ^${package}_ $UPSTREAM) -upstream=$( higher $LIST) - -if dpkg --compare-versions $current lt $upstream 2>/dev/null -then - - # echo Current version in $DISTRO: $current 1>&2 - # echo Upstream version in $UPSTREAM: $upstream 1>&2 - - echo Current version in $DISTRO: $current - echo Upstream version in $UPSTREAM: $upstream - [ $current = "EOL" ] && continue - if [ $package = "grub2" ] - then - echo WARNING! grub2 packages need to be manually compiled! - echo - continue - fi - - cd $WD - $TEST git checkout $DISTRO - $TEST git merge master - cd $TMP - echo Upgrading $current at $DISTRO with ${UPSTREAM}_$upstream 1>&2 - echo Running makepackage $package $DISTRO at $(date) $LOGS/$DISTRO-$package-$(date +%Y%m%d).log - echo Running makepackage $package $DISTRO at $(date) $LOGS/$DISTRO-$package-$(date +%Y%m%d).log 1>&2 - $TEST sh $WD/makepackage $package $DISTRO > $LOGS/$DISTRO-$package-$(date +%Y%m%d).log && echo -n Done! || echo -n FAILED! - echo " ($(date))" - echo -fi -done -} - -update - -for pair in $PAIRS -do -DISTRO=${pair/-*/} -UPSTREAM=${pair/*-/} -PACKAGES=$(git --git-dir=/home/systems/devel/.git ls-tree -r --name-only $DISTRO|grep helpers/make-|sed 's/.*make-//') -check >> $FILE -done - -report () { -#address=trisquel-devel@listas.trisquel.info -address=ruben@trisquel.info - -echo "Subject: Compiling updates -To: $address - -$(cat $FILE)" | $TEST /usr/sbin/sendmail -f watchdog@devel.trisquel.info -h correo.sognus.com -t $address -} - -grep -q makepackage $FILE && $TEST report -rm $FILE $TMP -rf