Use .dmg rather than .zip for Mac OS X

added component::applications/tor bundles/installation owner::hellias priority::high resolution::fixed status::closed tbb-3.0 tbb-easy tbb-usability type::task labels

So I tinkered a little bit with different compression algorithms and different hdiutil commands and in ended up with these conclusions.

The Tor Browser Bundle compressed to .dmg with the following command:

  hdiutil convert TorBrowserBundle.dmg -format UDZO -imagekey zlib-level=9 -o TorBrowserBundle-compressed.dmg

has a total file size of 3.2364 KB.

This is compressed using zlib level of 9, so optimized memory consumption minimized speed.

If I compress the original bundle using bzip2 I actually end up with a smaller file than the current .zip: 2.9976 KB.

So other than file size, what are the constraints? It seems like file size is taken care of in all cases here?

Trac fail. Erinn didn't see the mails for this bug. Wrong owner/component and wrong CC (her trac username is erinn).

Trac:
Owner: mikeperry to erinn
Component: Tor Browser to Tor bundles/installation
Cc: helix hellais to erinn, hellais

Also, can you paste the bzip2 hdiutil command? It looks like it is UBDZ instead of UBZO, and maybe another switch or two for max compression level?

We were chatting on IRC Mike - so Erinn has seen the bugs.

Trac:
tor-browser-bundle-OSX-dmg.diff

Patch for making TBB on OS X a .dmg

I have made a patch for the Makefile to build the TBB for OSX. I haven't tested it in the whole process, but I think it is correct.

You need to place TorBundleTemplate.dmg.bz2 in the scripts folder to make it work.

I am unable to shrink the template file down to less than 2MB. So I can't upload it here. If you tell me where I should place it I will upload it there (it is around 30MB).

The template file is what is required to have the background image and the correct icon positioning etc.

Here's the screenshot for the DMG: http://postimage.org/image/1xqc4q8kk/

About the 25MB limit on GMail: I can only assume that its sort of a soft-limit and the real hard limit is somewhat above 25MB. Otherwise the current TBB package for Windows wouldn't make it through.

I really like the dmg screenshot. I have also run into OS X users confused by the .zip file, fwiw. We should definitely be shipping a .dmg, esp if we can make it bzip2 and smaller than a .zip.

Well all around, I think this is an improvement; helix, do you need anything else?

hellais, great work and thanks a lot for this patch! I am going to make some changes to it and the template instead of accepting as-is, which I will post here once I am done. (It is probably just going to be things like separating the whitespace patch and tweaking the workflow to fit with how TBB is built.) Do you have some web space where you can upload the template? If so and you do not want to share the link here publicly, can you send it to me privately?

Here's how Chrome does things for their dmg: https://src.chromium.org/viewvc/chrome/trunk/src/chrome/installer/mac/pkg-dmg?view=markup

Also some notes on the DS Store File Format: https://wiki.mozilla.org/DS_Store_File_Format

Trac:
update_dsstore.applescript

chrome apple script

Also check out the file (update_dsstore.applescript) that I just uploaded.

Sorry for the delay in uploading and updating with the .dmg template, but the connectivity that I had in the past few days didn't allow me to upload the file :(.

Anyways, now you can find it here: http://server.globaleaks.org/TorBundleTemplate.dmg.bz2

sha512sum TorBundleTemplate.dmg.bz2 2351fcf18838e3b9da72c290d8962aa4c60ae89f1e40a2b4d7510902f372a5b39d4f7759f00c5377214b34f0661654445aae60e5897a57a9b88ef54ba9afeac1 TorBundleTemplate.dmg.bz2

shasum TorBundleTemplate.dmg.bz2 1303ee0de238101793c255b55cc8a4fb7e70e7f6

and to make also rransom happy:

md5sum TorBundleTemplate.dmg.bz2 d95e71f2cce824d61270885aa6648d22 TorBundleTemplate.dmg.bz2

:D

hellais,

Could you please look at the way Chromium and Vidalia do it (https://gitweb.torproject.org/vidalia.git/tree/HEAD:/pkg/osx) and try to use a more programmatic method? I don't think the template solution works because in order to make it replicable for other people, the template would have to be in git, which is a definite no-go. Also how did you generate the .DS_Store file? Am I correct that the template mainly just has a .stuff/tor-browser-bundle.png and a .DS_Store? If so we ought to be generating the .dmg from the .app, rather than using the template solution.

I have made a patch for making this work programmatically. Creating DS_Store files is a PITA. What mozilla does is even more dirty than the solution that we are going to use. The basically take a .DS_Store file and modify it to their needs.

What we will do is instead use AppleScript to set the parameters of the .dmg image and have the OSX handle the DS_Store writing.

On a high level what it does is create a .dmg image with background image, icons and proper sizing.

The only doubt that we had with erinn is that this will have a slightly different behavior than what users are usually used to. Since the .dmg is mounted as read-only they will not be able to run TBB directly from the .dmg container.

We have solved this by writing in the background image of the .dmg the text "Copy to you applications folder or to an external drive".

This is how the mounted image looks like: http://hellais.github.com/drop/Tor%20Browser%20Bundle%20OSX-screenshot.png

Trac:
osx-dmg.patch

DMG writing for Tor Browser Bundle

Oh this is very close! Thanks a lot. Here are some immediate (and relatively minor) suggestions:

Can you make it so that it follows the naming convention for the Mac OS X browser bundles? These are determined in the makefile (torbrowser.git/build-scripts/osx.mk in the compressed-bundle-localized target, such as: zip -r
$(DISTDIR)/$
(DEFAULT_COMPRESSED_BASENAME)$(LANGCODE).zip $There was an error rendering this math block. KaTeX parse error: Expected group after &#39;_&#39; at position 7: (NAME)_̲$ (LANGCODE).app)

You can probably reconstruct the basic name format by looking at the standard bundle name right now (TorBrowser-2.2.35-4-dev-osx-i386-ar.zip ), or we can change it. The information that must be contained, as per the naming conventions mentioned in torbrowser.git/docs/HACKING, is the version, arch, and language. I think something like TorBrowser-2.2.35-4-osx-i386-ar.dmg should be okay. It occurs to me that I don't mention the langcode in the HACKING doc, so I'll have to update it...

Where is the applescript we talked about?

Regarding the image, I think it's a good start. Would you mind me emailing Jeremy to see if he has any suggestions?

Thanks again, we're really close now!

Trac:
Cc: erinn, hellais to erinn, hellais, g.koppen@jondos.de

Every time I download the .zip on my Mac, I'm a little saddened this isn't merged yet. Is this just blocking because no one is aware who is supposed to make the tweaks? I'm going to assign this to hellias and put it in needs_revision in the hopes that that is all it takes :).

Trac:
Owner: erinn to hellias
Cc: erinn, hellais, g.koppen@jondos.de to erinn, hellais, g.koppen@jondos.de, Sebastian, ioerror
Status: new to assigned

Trac:
Status: assigned to needs_revision

No, this isn't merged because it doesn't make sense. .dmg files are for applications which get dropped into /Applications, which TBB should never be on any system with more than one user. People who separate an admin account from their normal account (like sane people should) are always surprised when their TBB doesn't work right for them if they put it int /Applications, or C:\Program Files on Windows, etc.

Also note that for the computer illiterate, the default behaviour of Safari is to "open safe files after downloading", which means people will automagically have the extracted file in their download directory, already extracted. No .zip file wrangling necessary. I absolutely agree that .dmg is the proper way to distribute normal applications on osx, but TBB is not a normal application like all the others. We could fix that, but by doing so we'd need to accept that there's no way to confine TBB to one single directory anymore. That's probably what we should do anyway, as the portability aspect is much less important now that we force TBB for everyone rather than have the possibility of the Vidalia bundle; but that's a separate discussion (which we should have, asap).

Replying to Sebastian:

I absolutely agree that .dmg is the proper way to distribute normal applications on osx, but TBB is not a normal application like all the others. We could fix that, but by doing so we'd need to accept that there's no way to confine TBB to one single directory anymore. That's probably what we should do anyway, as the portability aspect is much less important now that we force TBB for everyone rather than have the possibility of the Vidalia bundle; but that's a separate discussion (which we should have, asap).

‘trams’ reports that TBB is not confined to one directory on MacOS. I don't see any hope of fixing the ooze on Mac, so it probably is time to give up there.

Replying to Sebastian:

Also note that for the computer illiterate, the default behaviour of Safari is to "open safe files after downloading", which means people will automagically have the extracted file in their download directory, already extracted. No .zip file wrangling necessary.

Can downloading another zip archive cause Safari to unpack other random files into the TBB directory?

Replying to rransom:

‘trams’ reports that TBB is not confined to one directory on MacOS. I don't see any hope of fixing the ooze on Mac, so it probably is time to give up there.

I think it's way too early to conclude that we should give up there, especially since trams is just starting to get results. See also #5791 (moved).

Replying to Sebastian:

Also note that for the computer illiterate, the default behaviour of Safari is to "open safe files after downloading", which means people will automagically have the extracted file in their download directory, already extracted. No .zip file wrangling necessary. I absolutely agree that .dmg is the proper way to distribute normal applications on osx, but TBB is not a normal application like all the others. We could fix that, but by doing so we'd need to accept that there's no way to confine TBB to one single directory anymore. That's probably what we should do anyway, as the portability aspect is much less important now that we force TBB for everyone rather than have the possibility of the Vidalia bundle; but that's a separate discussion (which we should have, asap).

I am unable to think like either an osx user or windows user wrt installers. So while I guess people look to me as good at simulating users in general, in this case I'm not useful.

Do we have among us the right people with the right skillsets to both figure out what the packages should be doing on each platform, and change them so they do it? Or should we be trying to find those people?

Pretty much every other OSX app in the world has a .dmg that has the .app folder and a shortcut to Applications folder, and an arrow or some other instructions that indicate you need to drag one to the other.

So how about instead of having an Applications folder shortcut, we just have a shortcut to the Desktop folder? Wouldn't that solve the problem? Or does this drag operation itself update all sorts of plists, regardless of the destination folder?

I know for me, I hate the zip file because it naturally downloads to Downloads, and then I click on it (if you download TBB from TBB, it does not auto-unzip), and then the TBB app is in the wrong place (the Downloads folder still). Worse, when I download more than one update, the app gets renamed to things like "Tor Browser-en_US 2" by default after clicking on the .zip. This is a nightmare, because a non-zero number of novice users will just continue to run the old, exploitable Tor Browser-en_US from Downloads, if they even manage find the damn thing in Downloads in the first place.

Replying to Sebastian:

No, this isn't merged because it doesn't make sense. .dmg files are for applications which get dropped into /Applications, which TBB should never be on any system with more than one user. People who separate an admin account from their normal account (like sane people should) are always surprised when their TBB doesn't work right for them if they put it int /Applications, or C:\Program Files on Windows, etc.

If the issue is the fact that the application ends up installed for all the users of the system instead of putting an icon shortcut to /Applications we can put an icon shortcut to ~/Applications. This is done by a few applications on OSX and it's the equivalent of doing an installation of an app only for that user.

Also note that for the computer illiterate, the default behaviour of Safari is to "open safe files after downloading", which means people will automagically have the extracted file in their download directory, already extracted. No .zip file wrangling necessary. I absolutely agree that .dmg is the proper way to distribute normal applications on osx, but TBB is not a normal application like all the others. We could fix that, but by doing so we'd need to accept that there's no way to confine TBB to one single directory anymore. That's probably what we should do anyway, as the portability aspect is much less important now that we force TBB for everyone rather than have the possibility of the Vidalia bundle; but that's a separate discussion (which we should have, asap).

What is the difference with running TBB from ~/Applications compared with running it from your ~/Downloaded directory? In the second it just pollutes a directory that is uber polluted already and the user actually looses track of it.

It's also worth noting that the default behavior in Safari for unzipping files automatically makes you end up with the .zip file inside of your trashcan. The user may not be aware of this and could lead him into getting into trouble.

FYI: http://petsymposium.org/2012/papers/hotpets12-1-usability.pdf lists "archive confusion" as the 4th most common "stop issue" that prevented people from using TBB. They tested only Windows 7, which uses the self-extracting exe, which I would have actually guessed as easier to figure out than a zip? Even in the best case, it's the same user activity though: Click on exe/zip, lose track of extraction folder, get confused.

Trac:
Keywords: N/A deleted, tbb-usability added

Trac:
Cc: erinn, hellais, g.koppen@jondos.de, Sebastian, ioerror to erinn, hellais, g.koppen@jondos.de, Sebastian, ioerror, gnorcie@soic.indiana.edu

Replying to hellais:

Replying to Sebastian:

No, this isn't merged because it doesn't make sense. .dmg files are for applications which get dropped into /Applications, which TBB should never be on any system with more than one user. People who separate an admin account from their normal account (like sane people should) are always surprised when their TBB doesn't work right for them if they put it int /Applications, or C:\Program Files on Windows, etc.

If the issue is the fact that the application ends up installed for all the users of the system instead of putting an icon shortcut to /Applications we can put an icon shortcut to ~/Applications. This is done by a few applications on OSX and it's the equivalent of doing an installation of an app only for that user.

Also note that for the computer illiterate, the default behaviour of Safari is to "open safe files after downloading", which means people will automagically have the extracted file in their download directory, already extracted. No .zip file wrangling necessary. I absolutely agree that .dmg is the proper way to distribute normal applications on osx, but TBB is not a normal application like all the others. We could fix that, but by doing so we'd need to accept that there's no way to confine TBB to one single directory anymore. That's probably what we should do anyway, as the portability aspect is much less important now that we force TBB for everyone rather than have the possibility of the Vidalia bundle; but that's a separate discussion (which we should have, asap).

What is the difference with running TBB from ~/Applications compared with running it from your ~/Downloaded directory? In the second it just pollutes a directory that is uber polluted already and the user actually looses track of it.

It's also worth noting that the default behavior in Safari for unzipping files automatically makes you end up with the .zip file inside of your trashcan. The user may not be aware of this and could lead him into getting into trouble.

A typical user expects TBB to act like an application. As long as they can navigate to their "Applications" folder, and click on something that launches the TBB, they will be happy.

I understand there are technical constraints here, but users have a preconceived notion about how software installation occurs. They may not be familiar with concepts like a user directory vs a system directory.

If having TBB installed systemwide is an issue, by all means, avoid it at all costs.

But we should have the installation process be as similar to any other OSX application.

Trac:
Username: greg

What's the status here? Is it blocking on something?

I want this for TBB 3.0, but we now build MacOS binaries on Linux (using Gitian), so this patch won't work there. We need to do this instead: http://stackoverflow.com/questions/286419/how-to-build-a-dmg-mac-os-x-file-on-a-non-mac-platform#3776463

That post doesn't mention how to make it use bzip2, though (which we want).

Trac:
Priority: normal to major
Keywords: N/A deleted, tbb-3.0 added

It turns out that stackoverflow link does not successfully create UDIF/UDBZ compressed dmg images (they produce CRC errors when used on MacOS). However, it looks like Mozilla succeeded at this by forking libdmg-hfsplus to add UDBZ support. See https://bugzilla.mozilla.org/show_bug.cgi?id=935237 and https://github.com/vasi/libdmg-hfsplus/tree/bzip2. A LZ compressed DMG (UDZO format) can also be produced with https://github.com/erwint/libdmg-hfsplus, but it will likely be larger than UDBZ.

If their UDBZ fork actually works as they claim, this should be relatively easy, though perhaps a little time consuming to build, test, and get the shortcuts to work properly to provide an easy link to the Desktop and/or the Applications folder for maximum usability.

Trac:
Keywords: N/A deleted, tbb-easy added

The following seems to more or less work:

# First: Extract TBB + localization files in ~/install/tbb/
# Then do:
mkdir -p ~/install/tbb/.background/
cp ~/build/tor-browser/browser/branding/official/background.png ~/install/tbb/.background/
cp ~/build/tor-browser/browser/branding/official/disk.icns ~/install/tbb/.VolumeIcon.icns
cp ~/build/tor-browser/browser/branding/official/dsstore ~/install/tbb/.DS_Store
ln -s /Applications "~/install/tbb/ "
genisoimage -D -V "TorBrowserBundle_en-US" -no-pad -R -apple -o tbb-uncompressed.dmg ~/install/tbb
./dmg/dmg dmg tbb-uncompressed.dmg tbb-compressed.dmg

However, the background and the icon positioning is not properly picked up, possibly because the provided dsstore file still references Firefox instead of TorBrowserBundle_en-US. A simple sed -i to change those strings did not appear to work. I tried asking in https://bugzilla.mozilla.org/show_bug.cgi?id=935237#c32. We'll see what Mozilla says about how we can generate/edit dsstore.

This was fixed in 3.6-beta-1. It's also reproducible, with an additional patch to libdmghfs-plus.

Trac:
Status: needs_revision to closed
Resolution: N/A to fixed

closed

mentioned in issue #7842 (closed)

mentioned in issue #8119 (moved)

mentioned in issue #10104 (closed)

Use .dmg rather than .zip for Mac OS X

Child items ...

Activity