14 May 2010

How to save your rpm's for future reuse in Mandriva.

How to save your rpm's for future reuse in Mandriva. By Glenn Waller 28th April 2010.

Why did I decide to reuse the downloaded rpm's (Redhat Package Management)?

To save time, connectivity problems (Internet) Bandwidth and embarrassment after I broke the system. I have added my scripts with explanations to the end of this paper but they are customised for my system, so read and edit them before trying to use them. ;-)

How I started reusing the saved rpm's.

Including, Mandriva installation choice of copying the data to HD, before the package selection process, was saved in /var/ftp/pub/mandriva(ver)...

Mostly, back then it was for large packages that I would reuse, such as the kernel-source (50Mb, approx) and the plf (Mandriva's Penguin Liberation Front) rpm's for video codecs.

Then I found genhdlist and soon after genhdlist2, and wrote a bunch of commands to a text-file and made a shell script out of it.

I partitioned my hard-drives to give me some space that would not be formatted during an install, Mandy generally requires formatting of /usr and / (root). This maybe a GNU/Linux requirement across the board of the many distributions (distros)

After I found genhdlist2 was I able to usefully add my local repositions (repos) to drakRPM, to make them available when using smart-packager and urpm(i), And reuse them at will, regardless of when I broke it and, and keep moving forward.

Using the no-clean hack in the GUI has never been reliable on my system, as it seems to reset it's self after each invocation of the program. So I took a different track, I wanted to save the rpms while at the same time being able to view the selected packages and dependencies (deps) before committing to the changes.

So, I would painstakingly copy and paste the confirmation text from Drakrpm to a plain text file and remove the new-line and dash+space at the beginning of each new line to create a space separated list that urpmi could work with. After the copy and paste and before running the list with urpmi –noclean I would close Drakrpm cancelling the process, but my list of packages was safe, and usable.

I still have not figured out how to keep the signatures for the packages, although I have tried using some guides from the web. But I guess if they were fine when initially downloaded, they'll still be fine unless I get a hard-drive error, which has not occurred for quite some time (Fingers crossed).

The list was generally discarded after use, generally because of the gross amount of redundant files I would end up with, sometimes it would take upto an hour to edit the list (1340 odd packages, lines) so I've been hunting for a way to make this more expedient and reliable. And I found sed.

Editing the list was accomplished from the bottom to the top, so I was not continually working at the end of the line, but the beginning of the line to save scrolling, with the mouse to high-light the characters I wanted/needed to remove and the space bar.

“Two hands, two guns!” (sifu Malcolm Sue)

Now with all of that it only takes a second to convert the list and I can enjoy my time doing what ever else I like.

I hope someone else finds this helpful, cheers and kind regards, Glenn (GlennsPref@gmail.com)

Scripts with explanations....

The first was a backup script which only collected the downloaded rpms from /var/cache /urpmi/rpms to a directory safe within /home/glenn/....I have many partitions, don't know why, Maybe just because I can, lol.

glenn@GamesBox:~$ df

Filesystem Size Used Avail Use% Mounted on

/dev/sda5 20G 1.8G 17G 10% /

/dev/sda1 92M 59M 28M 68% /boot

/dev/sda14 383G 232G 151G 61% /home

/dev/sdc7 33G 176M 33G 1% /home/glenn/local/Music

/dev/sdd5 52G 180M 52G 1% /home/glenn/local/Video

/dev/sdb8 73G 61G 13G 84% /home/glenn/local/spare

/dev/sdc1 56G 180M 55G 1% /home/glenn/local/stuff

/dev/sdb7 34G 31G 2.9G 92% /home/glenn/local/vm1

/dev/sdb9 74G 51G 23G 69% /home/glenn/stuff

/dev/sdb1 22G 13G 9.4G 57% /media/win_c

/dev/sdd6 31G 14G 18G 44% /media/win_c2

/dev/sdb6 29G 19G 9.8G 66% /media/win_d

/dev/sdd7 63G 23G 41G 36% /media/win_d2

/dev/sda11 5.0G 138M 4.6G 3% /opt

/dev/sda8 8.0G 157M 7.4G 3% /tmp

/dev/sda7 9.9G 5.3G 4.1G 57% /usr

/dev/sda10 5.0G 139M 4.6G 3% /usr/local

/dev/sdc6 20G 520M 18G 3% /usr/src

/dev/sda6 20G 911M 18G 5% /var

/dev/sdc5 3.5G 72M 3.3G 3% /var/ftp

/dev/sda13 5.0G 139M 4.6G 3% /var/spool

/dev/sda12 2.0G 56M 1.9G 3% /var/www


That is all it did.

Of course to simplify the editing of this script, it calls other scripts that do most of the work,

and it's still a “work in progress”(wip). It has some text to put up on the screen to give some

feedback to the user of what stage the process is up to, and although some of them are duplicated in the other scrips, it works quite well (IMHO).

#! /bin/zsh

#echo "/home/glenn/bin/backup-rpms"

echo "/home/glenn/bin/backup-rpms"


sudo sh /home/glenn/build/rpms/organise-rpms

echo "call organise-rpms....."

echo "Backup-rpms"

#echo "generating new hdlist.cz for local archives........"

sudo sh /home/glenn/build/rpms/gen-hdlist2


echo "Backup-rpms"

echo "rebuilding local rpm and file databases... this might take a while ........"

sudo rpm --rebuilddb && sudo updatedb


echo "Backup-rpms"

echo "local database up to date ........"



sudo sh /home/glenn/build/rpms/update-rpm-database

echo "run rpm -qa | sort | less > /home/glenn/build/note-rpm-installed-list-`date +%Y-%m-%d-%H`.txt, maybe upto 15kb text."/n

sudo rpm -qa | sort | less > /home/glenn/build/note-rpm-installed-list-`date +%Y-%m-%d-%H`.txt

echo "run rpm -Va | grep miss & send to /home/glenn/build/anote-rpm-Va-grep-miss-list-`date +%Y-%m-%d-%H`.txt"

sudo rpm -Va | grep miss > /home/glenn/build/anote-rpm-Va-grep-miss-list-`date +%Y-%m-%d-%H`.txt

echo "clean rpm cache"

sudo smart clean

sudo rm -rf /var/cache/urpmi/rpms/*rpm

echo ""

echo "all repositories are now up to date, any errors? see above!"

date && whoami

The first thing it does is call organise-rpms from ~/build/rpms...

sudo sh /home/glenn/build/rpms/organise-rpms

This script finds rpms where they maybe stored by me, and the programs DrakRPM and

Smart. (I really like Smart, it is a synaptic-like package manager although it has been changed over

the years, and is now much more useful, but harder to hack, like saving the rpms (Grief)).[/code]

#! /bin/zsh

#echo "/home/glenn/build/rpms/organise-rpms"

echo "organise-rpms"

echo "Called by, ~/bin/backup-rpms"

echo "organising saved rpm cache for local archives....."

#1st Level

sudo mv -v /var/cache/urpmi/rpms/*2010.0*rpm /home/glenn/local/Archive/2010.0/noarch/

sudo mv -v /var/cache/urpmi/rpms/*2010.1*rpm /home/glenn/local/Archive/2010.1/noarch/

sudo cp -v /var/lib/smart/packages/*2010.0*rpm /home/glenn/local/Archive/2010.0/noarch/

sudo cp -v /var/lib/smart/packages/*2010.1*rpm /home/glenn/local/Archive/2010.1/noarch/

sudo mv -v /var/cache/urpmi/rpms/*rpm /home/glenn/local/Archive/2010.1/noarch/

sudo cp -v /var/lib/smart/packages/*rpm /home/glenn/local/Archive/2010.1/noarch/

sudo mv -v /home/glenn/Download/*rpm /home/glenn/local/Archive/2010.1/noarch/

sudo mv -v /home/glenn/Download/packagesNsource/*rpm

echo "filter for 2010.1 packages"

sudo mv -v /home/glenn/local/Archive/2010.1/noarch/*2010.0.x86_64.rpm /home/glenn/local/Archive/2010.0/x86_64/

sudo mv -v /home/glenn/local/Archive/2010.1/x86_64/*mdv2010.1.noarch.rpm /home/glenn/local/Archive/2010.1/noarch/

sudo mv -v /home/glenn/local/Archive/2010.1/x86_64/*plf2010*.rpm /home/glenn/local/Archive/2010.1/plf/

sudo mv -v /home/glenn/local/Archive/2010.1/noarch/*plf2010*.rpm /home/glenn/loca/Archive/2010.1/plf/

sudo mv -v /home/glenn/local/Archive/2010.0/noarch/*4.3.8*mdv2010.1.x86_64.rpm/home/glenn/local/Archive/2010.1/kde/

sudo mv -v /home/glenn/local/Archive/2010.0/noarch/*4.3.8*mdv2010.1.x86_64.rpm/home/glenn/local/Archive/2010.1/kde/

sudo mv -v /home/glenn/local/Archive/2010.0/noarch/*4.3.8*mdv2010.1.x86_64.rpm/home/glenn/local/Archive/2010.1/kde/

sudo mv -v /home/glenn/local/Archive/2010.0/kde430/*4.3.8*mdv2010.1.x86_64.rpm/home/glenn/local/Archive/2010.1/kde/

sudo mv -v /home/glenn/local/Archive/2010.0/kde430/*mdv2010.1.noarch.rpm /home/glenn/local/Archive/2010.1/noarch/

sudo mv -v /home/glenn/local/Archive/2010.1/x86_64/*4.3.8*mdv2010.1.x86_64.rpm/home/glenn/local/Archive/2010.1/kde/

sudo mv -v /home/glenn/local/Archive/2010.1/noarch/*4.3.8*mdv2010.1.x86_64.rpm/home/glenn/local/Archive/2010.1/kde/

sudo mv -v /home/glenn/local/Archive/2010.0/noarch/*mdv2010.1.noarch.rpm /home/glenn/local/Archive/2010.1/noarch/

sudo mv -v /home/glenn/local/Archive/2010.0/i586/*mdv2010.1.i586.rpm /home/glenn/local/Archive/2010.1/i586/

echo "cleanup-stagglers"

sudo mv -v /home/glenn/local/Archive/2010.1/noarch/*2010.1.x86_64.rpm /home/glenn/local/Archive/2010.1/x86_64/

sudo mv -v /home/glenn/local/Archive/2010.1/noarch/*2010.1.i586.rpm /home/glenn/loca/Archive/2010.1/i586/

sudo mv -v /home/glenn/local/Archive/2010.0/noarch/*4.3.8*mdv2010.1.x86_64.rpm/home/glenn/local/Archive/2010.1/kde/

sudo mv -v /home/glenn/local/Archive/2010.0/noarch/*plf2010.1.x86_64.rpm /home/glenn/local/Archive/2010.1/plf/

sudo mv -v /home/glenn/local/Archive/2010.0/noarch/*plf2010.0.x86_64.rpm /home/glenn/local/Archive/2010.0/plf/

sudo mv -v /home/glenn/local/Archive/2010.1/noarch/*mdv2010.0.noarch.rpm /home/glenn/local/Archive/2010.0/noarch/

sudo mv -v /home/glenn/local/Archive/2010.0/x86_64/*2010.0.noarch.rpm /home/glenn/local/Archive/2010.0/noarch/

sudo mv -v /home/glenn/local/Archive/2010.1/x86_64/*2010.1.noarch.rpm /home/glenn/local/Archive/2010.1/noarch/

sudo mv -v /home/glenn/local/Archive/2010.0/contrib/*.*mnb*.x86_64.rpm /home/glenn/local/Archive/2010.1/x86_64/

sudo mv -v /home/glenn/local/Archive/2010.0/noarch/*.*mnb*.x86_64.rpm /home/glenn/local/Archive/2010.1/x86_64/

sudo mv -v /home/glenn/local/Archive/2010.0/x86_64/*.*mnb*.x86_64.rpm /home/glenn/local/Archive/2010.1/x86_64/

sudo mv -v /home/glenn/local/Archive/2010.1/noarch/*.*mnb*.x86_64.rpm /home/glenn/local/Archive/2010.1/x86_64/

sudo mv -v /home/glenn/local/Archive/2010.1/contrib/*.*mnb*.x86_64.rpm /home/glenn/local/Archive/2010.1/x86_64/

sudo mv -v /home/glenn/local/Archive/2010.1/plf/*.*mnb*.x86_64.rpm /home/glenn/local/Archive/2010.1/x86_64/

sudo mv -v /home/glenn/local/Archive/2010.1/i386/*.*mnb*.x86_64.rpm /home/glenn/local/Archive/2010.1/x86_64/

sudo mv -v /home/glenn/local/Archive/2010.1/i586/*.*mnb*.x86_64.rpm /home/glenn/local/Archive/2010.1/x86_64/

sudo mv -v /home/glenn/local/Archive/2010.0/contrib/*.*mnb*.noarch.rpm /home/glenn/local/Archive/2010.1/noarch/

sudo mv -v /home/glenn/local/Archive/2010.0/noarch/*.*mnb*.noarch.rpm /home/glenn/local/Archive/2010.1/noarch/

sudo mv -v /home/glenn/local/Archive/2010.0/x86_64/*.*mnb*.noarch.rpm /home/glenn/local/Archive/2010.1/noarch/

sudo mv -v /home/glenn/local/Archive/2010.1/noarch/*.*mnb*.noarch.rpm /home/glenn/local/Archive/2010.1/noarch/

sudo mv -v /home/glenn/local/Archive/2010.1/x86_64/*.*mnb*.noarch.rpm /home/glenn/local/Archive/2010.1/noarch/

date && whoami

This script also calls another file to change the permissions so I can use them with out errors

such as read-only.

#! /bin/zsh

echo "Backup-rpms"

echo "~/build/rpms/rpms-chmod"

sudo chmod -R 755 /home/glenn/local/Archive/2009.0/

sudo chmod -R 755 /home/glenn/local/Archive/2009.1/

sudo chmod -R 755 /home/glenn/local/Archive/2010.0/

sudo chmod -R 755 /home/glenn/local/Archive/2010.1/

date && whoami

Next the script calls for genhdlist2 to recreate the files required for DrakRPM and Smart

package data-bases, hdlist.cz

#! /bin/zsh

echo "Backup-rpms"

echo "generating new hdlist.cz for local archives........"

echo ""

echo "Backup-rpms"

echo "generating new hdlist.cz for local archives........"

# echo "Backup-rpms local/Archive/2009.0/i386"

# sudo genhdlist2 --xml-info -v --allow-empty-media /home/glenn/local/Archive/2009.0/i386

# echo "Backup-rpms local/Archive/2009.1/noarch"

# sudo genhdlist2 --xml-info -v /home/glenn/local/Archive/2009.1/noarch

# echo "Backup-rpms local/Archive/2009.1/i586"

# sudo genhdlist2 --xml-info -v /home/glenn/local/Archive/2009.1/i586

# echo "Backup-rpms local/Archive/2009.1/i586/contrib"

# sudo genhdlist2 --clean --xml-info -v /home/glenn/local/Archive/2009.1/i586/contrib

# echo "Backup-rpms local/Archive/2009.1/i386"

# sudo genhdlist2 --xml-info -v --allow-empty-media /home/glenn/local/Archive/2009.1/i386

# echo "Backup-rpms local/Archive/2009.1/x86_64"

# sudo genhdlist2 --clean --xml-info -v /home/glenn/local/Archive/2009.1/x86_64

# echo "Backup-rpms local/Archive/2009.1/kde430"

# sudo genhdlist2 --clean --xml-info -v /home/glenn/local/Archive/2009.1/kde430

# echo "Backup-rpms local/Archive/2009.1/x86_64/contrib"

# sudo genhdlist2 --xml-info -v /home/glenn/local/Archive/2009.1/x86_64/contrib

echo "Backup-rpms 2010.0"

sudo genhdlist2 --clean --xml-info -v /home/glenn/local/Archive/2010.0/x86_64

sudo genhdlist2 --clean --xml-info -v /home/glenn/local/Archive/2010.0/noarch

sudo genhdlist2 --clean --xml-info -v /home/glenn/local/Archive/2010.0/i586

sudo genhdlist2 --clean --xml-info -v /home/glenn/local/Archive/2010.0/i386

sudo genhdlist2 --clean --xml-info -v /home/glenn/local/Archive/2010.0/contrib

echo "Backup-rpms 2010.1"

sudo genhdlist2 --clean --xml-info -v /home/glenn/local/Archive/2010.1/x86_64

sudo genhdlist2 --clean --xml-info -v /home/glenn/local/Archive/2010.1/noarch

sudo genhdlist2 --clean --xml-info -v /home/glenn/local/Archive/2010.1/i586

sudo genhdlist2 --clean --xml-info -v /home/glenn/local/Archive/2010.1/i386

sudo genhdlist2 --clean --xml-info -v --allow-empty-media /home/glenn/local/Archive/2010.1/contrib

sudo genhdlist2 --clean --xml-info -v /home/glenn/local/Archive/2010.1/kde

echo "Backup-rpms plf"

# sudo genhdlist2 --clean --xml-info -v /home/glenn/local/Archive/2009.0/plf

# sudo genhdlist2 --clean --xml-info -v /home/glenn/local/Archive/2009.1/plf

sudo genhdlist2 --clean --xml-info -v /home/glenn/local/Archive/2010.0/plf

sudo genhdlist2 --clean --xml-info -v --allow-empty-media /home/glenn/local/Archive/2010.1/plf

echo "GenHdList2 up to date!"

date && whoami

As new versions of Mandy come out, new lines are added and the old ones removed, as seen

above (#) commented out...

If genhdlist finds a corrupted package it stops and shows and error, naming the package,


Next it refreshes the rpm database of installed packages, and the find data-base too.

rpm --rebuilddb && sudo updatedb

This takes a while....

The next task is to update the external rpm repo data-bases....

#! /bin/zsh

#echo "/home/glenn/bin/backup-rpms"

echo "/home/glenn/build/rpms/update-rpm-database"

echo "Called from, ~/bin/backup-rpms"





echo "updating urpmi rpm database....."




#echo "urpmi.update local repos"

#sudo urpmi.update local

#echo "uncomment lines to run update all, skipped for now, GW"

echo "urpmi.update all"

sudo urpmi.update -a

#echo "smart update"

#sudo smart update


echo "Backup-rpms"

#echo "local repositories are up to date .. GamesBox.GlennsPref.net........Have a nice day!"

echo "urpmi is up to date .. GamesBox.GlennsPref.net........Have a nice day!"

date && whoami

smart update is no longer required, as it reads the URPM data-bases

The last things to run is a clean up, and checking for missing packages and a catalogue of

installed rpms for the system.

sudo rpm -qa | sort | less > /home/glenn/build/note-rpm-installed-list-`date +%Y-%m-%d-%H`.txt

echo "run rpm -Va | grep miss & send to /home/glenn/build/anote-rpm-Va-grep-miss-list-`date +%Y-%m-%d-%H`.txt"

sudo rpm -Va | grep miss > /home/glenn/build/anote-rpm-Va-grep-miss-list-`date +%Y-%m-%d-%H`.txt

echo "clean rpm cache"

sudo smart clean

sudo rm -rf /var/cache/urpmi/rpms/*rpm

rpm -Va | grep miss, takes a while to finish, but can be well worth the wait, before rebooting and

having a crash or cli only system because of a missing dep or lib.

The output is redirected to a file and the file name is created with the date and hour it was


Now the reason I written this is because of a sed trick I found that converts the list to

something urpmi can work with, This process is not completed yet, more things will occur to me as time rolls on,


echo "Create a space-separated-list from top-down list with preceeding chars."

cd ~/build

echo "remove old files"

rm -f ~/build/file2.txt

rm -f ~/build/file3.txt

echo "Remove - (dash space) from lead of lines using sed...."

echo "sed 's/^[-\ ]*//' file1.txt (from file1.txt in pwd dir)"

echo "redirect output to new file...file2.txt"

sed 's/^[-\ ]*//' file1.txt > file2.txt

echo "Now replace carriage returns with a space... using file2.txt, create file3.txt"

sed ':a;N;$!ba;s/\n/ /g' file2.txt > file3.txt

First it removes the old files, to save stopping for errors, file exists.

Then it removes the dash space from the head of the lines...

And lastly it replaces the newline char with a space, bringing all the list to one line, sweet!

I'd like to reference where most of my info came from,

MDE repo http://www.mde.djura.org/index.html and

SoS repo http://seerofsouls.com/ (website down atm), help files...

Few years old now but still very useful.



And that's IT!

Any questions? Did I miss something? Please ask!

Regards Glenn.

No comments:

Post a Comment