zdrv evolution

I am continuing with a project that I have been gradually preparing the groundwork for.

Late last year, maybe September/October/November, I built an experimental Puppy with no zdrv file. Instead, all the modules were in the initial ramdisk. I then moved them all across to the main filesystem before the switch_root -- so they ended up getting saved in the pup_save file.

There were some stumbling blocks with that concept. One was the internal format of the zdrv file itself -- but a few months ago, in one of the 4.00 alphas, I redesigned the layout of the zdrv file, such that the firmware packages were kept in /lib/all-firmware directory. This was one stumbling block removed in my plan to return to putting the zdrv file inside the initial ramdisk (initramfs).

Right now I'm running another such experimental build. The modules are all inside 'initrd.gz', so it is mighty big -- 20MB. The modules are all compressed, so it doesn't get much bigger than that when loaded into RAM.

The big advantage is that the complete set of modules are present right at that first stage of booting, including the massive complete set of Sage's treasured SCSI drivers.

All the modules are moved to the main filesystem just before the switch_root, so they are all there when the boot scripts are running.

Another thing that has motivated me to return to this idea, is the automatic module probing performed by the kernel. The kernel probes, that is, calls modprobe, typically passing an alias, querying whether there is a matching module. The problem with this is that modprobe is only able to match an alias to a module if the module is already present. The fetching system from the zdrv file does not work in this situation.
Note, having all the dependency files does not help, the actual module has to be there for modprobe to match an alias to a module.

Having all modules present greatly simplifies the /sbin/modprobe script. Which I have done. The script still does the job it was mainly written for, to load the firmware of a module the first time it loads.

The question then is, what to do with the modules after that? We probably don't want to leave them taking up 18MB of RAM or pup_save space. This is what I have been contemplating. The experimental build I'm running right now has them sitting in the pup_save file.

There are various possibilities:

1. One somewhat inflexible solution is to simply delete them all after it has been decided all required modules are loaded. Not so good if you want to load another module after bootup.

1b. Same as above, except if a module is requested post-bootup, it can be registered as required in (say) /etc/MODULESCONFIG and you are then asked to reboot Puppy and the boot scripts will load it. A bit round-about but workable if such post-bootup modules are a rarity -- well it would be a once-off thing for each module anyway.

2. A boot script can create a zdrv file, at /mnt/home/zdrv_xxx.sfs, with all modules and firmware. The modprobe script can still fall back to fetching from a zdrv file. So, Puppy has easy access to the complete set of modules during bootup, and if any stray module is needed after bootup then modprobe will fetch it from the zdrv file in the normal way of previous puppies.

3. Just leave them all in the pup_save file. very convenient, and 18MB is not much space if you have a 1GB pup_save.

3b. Leave them in the pup_save, except delete a big chunk of them that are unlikely to be needed after bootup -- which would be the majority.

A note about something related: the idea of mounting and unmounting zdrv as a unionfs layer, on-the-fly. I have played with that, and found it very troublesome.

---

Comments

Does this make sense?
Username: Bosco Bearbank
I don't know about we, but I really don't want to leave them taking up 18MB of RAM or pup_save space. My preference, if possible, is option 2b: A boot script can create a zdrv file, at some user-unmountable place, with all modules and firmware...? This combines the best of 1 and 2 without needlessly taking up RAM or pup_save space.

zdrv
Username: kirk
"I like option 3, copy it to the pup_save file. 18MB is just not that much in a pup_save file and I doubt many users run RAM-only on low RAM computers. Simple solutions lead to less problems. How much does the big initrd effect your boot time? A side note about copying the pup_4xx.sfs to ram. Unless the pup_4xx.sfs file is on CD or we have LOTS of RAM, probably should not copy to RAM. Really need the RAM to run programs.

Good idea :-)
Username: pakt
"Barry, thanks for sharing your ideas with us. Option 3 sounds best to me - leave the modules in the pup_save file. This must be the simplest, most efficient solution. The only gotcha would be when running in RAM - with no pup_save file the modules would have to stay in RAM. Otherwise, move the modules to pup_save directly after boot to free RAM. Besides making Sage happy ;-), it would also solve a minor inconvenience I have. I have a PCMCIA USB adapter in my laptop replacing a defective USB port. My USB keyboard and mouse are connected to this adapter. Since the PCMCIA module is not in initrd.gz but in rc.modules which runs later on, my USB keyboard cannot be used to choose a pup_save file. Your solution would also allow booting (USB harddisk, USB CD/DVD drive, Flash drive, etc) through a PCMCIA USB adapter ;-)

Zdrv in nitramfs
Username: BarryK
"Yes, having the complete set of modules in the initramfs means we can boot from anything, no worries about whether the correct module is there or not. Someone mentioned to me awhile back that they had experimented with mounting the zdrv file in the initramfs -- that is, it's still a file. Who was that? But that does take us back to that catch-22 situation, where the initramfs may need a special driver to access the boot device. Yes, it takes longer to load initrd.gz. On the first boot, this is compensated by the modules loading quicker.

Option 3c
Username: BarryK
"There is another option, 3c. Put the modules into the pup_save, but then progressively delete them as the pup_save gets full. The daemon that monitors free space can do that. The free memory applet in the taskbar could even take them into account, calculate the free space, add on 18MB (or whatever), so the user won't even know that they are taking up space. How about that for an interesting solution? Then the init script in the initiramfs can restore them if there is sufficient free space in the pup_save!

Option 3d
Username: jamesbond
"3c is definitely interesting. How about doing it manually instead? Leave them in pup_save. But give a shortcut / button / menu-item to: 1. delete all the excess modules from pup_save 2. restore the modules from initramfs (after reboot) back to pup_save if (1) is accidentally executed. With this, those who doesn't like it can delete it (e.g. ppl running in RAM etc), those who prefer it will have it by default, and those who accidentally / mistakenly delete it can recover. Complexity-wise, I think it's simpler this way than 3c, lesser chance of bugs doing something funny behind our back :)

3c
Username: urban soul
"Spending the whole day hacking pwireless.. these are welcome fundamental thoughts. Don't get away from treating the user as 'major' or 'root' or fully responsible or whatever. Foolproof design can be achieved nevertheless. I vote for option 3b with a transition phase beginning with option 2... (because 3b needs some intelligent decisions which may become known later on)

zdrv in intramfs
Username: crafty-one
"@Barry Needed to change my login to this blog - last one won't let me in..!! zdrv - I think 3c is workable - however I also think you need to possibly look at a script that tries to detect what type of motherboard/processor combo the user is loading Puppy into - then remove from those modules from zdrv that will never be used for the users' motherboard/processor combo..! e.g.. my combo= VIA m/b with AMD Duron CPU.. so won't need to keep any INTEL or ARM or PPC - etc.. modules in the zdrv..! Hopefully you get my drift with this..? crafty. . .

Boot from USB
Username: CHLee
"I like option 3d. I only worry booting time too long for USB boot, currently the initrd.gz plus vmlinuz size only 2MB. My PC need about one minute to load this two files. If the initrd.gz size 18MB, then need 9 minutes to load them. Can we speed up the loading time? Thanks Barry!

Boot from USB
Username: CHLee
"Correction! The initrd.gz plus vmlinuz size are 3MB. So,18MB initrd.gz need 6 minutes. Sorry!

Boot from USB
Username: GeorgR
"I like option 3a. After first experiences we will see whether we need to optimize memory usage with 3b or 3d. I like the chance for better hardware recognition and perhaps even speeding up overall booting time. But I see the point of CHLee and the risk of long booting times on mainboards with bios handling usb ports only at slow speed even with usb 2.0 chipsets. Lets test it!

rebuild initramfs
Username: Raffy
"(Don't know initramfs :) but here's my 002c. Over 90% of the time the basic kernel modules do the trick and load other drivers successfully. For this, we can keep the status quo. However, for the exceptional cases, could the boot script that encounters a "no-driver" situation produce a report of what hardware are without ready drivers? Then the user can shop for supplemental init-PETs that contain his/her needed drivers? Easy remastering and package creation are some of Puppy's strengths. Maybe we can apply this to distribute supplemental initramfs packages? For example, for rebuilding your initrd.gz?

zdrv modules
Username: tronkel
"How about making a list of the modules actually used and then merging them with the pup_xxx.sfs file on the fly, if requested by the user. Then the big zdrv.sfs can be deleted. Next time the system boots, it could look at the pup_XXX.sfs and use the same modules as previously.

removing the zdrv
Username: HairyWill
"I'm not sure how Tronkel's solution above would work. Squash files are readonly. How about an option to rewrite initrd.gz after installing to reduce boot time. If the user choose to they could replace an installed initrd.gz with a version that only contained the drivers for the system they had installed on. It would also be possible for something like a pendrive install to have multiple copies of initrd.gz with different driver sets for different hardware.

sfs
Username: pizzasgood
""editing" a .sfs file involves just building a new one. For zdrv_xxx.sfs that doesn't take very long, but it can be really slow for pup_xxx.sfs, and for large derivatives of Puppy it can take over fifteen minutes. The other problem with modifying pup_xxx.sfs, or initrd.gz for that matter, is that if the user was booting from CD, he'd have to burn a new disk. I do like the idea of an "optimize my initrd.gz" button though, for the people who have an hd or usb install, or don't mind burning another disk (or the same one if rw). But that's just a happy extra, not a solution, IMHO. I like either jamesbond's idea of letting the user opt to delete/restore the unused modules, or else just keeping them all in the pup_save.2fs file. They're nice and simple, and don't involve things happening behind the user's back. The idea of having Puppy delete modules incrementally would lead to never being sure exactly what modules are installed, or how much space you really have (as though THAT problem needs to be expanded...).

zdrv
Username: tronkel
"Although sfs files are read-only, a script could mount and open them and then add the reduced module set. They could be held if necessary in a temporary pup_save file. The script could then re-squash the amended sfs file. Something along the lines of MU's SFS-Combiner script. Then the temp files could be deleted Since it could be somewhat slow operation, the user could have the choice to perform this task at will - by clicking on an option in the WizardWizard for example.

PC Profile
Username: SouthPaws
"Hey Barry...Keep up the good work! HairyWill touched upon something I've wondered about. Would it hinder or hurt for Puppy to yield two save files, one being specifically hardware related: a PC-Profile along with the pupsave... All your data in the pupsave...could be used worry-free with multiple pcs... Plug in your usb drive, choose your profile...home,work,friends pc,test pc,etc...and off you go!!!

Tags: puppy