External Imports

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

External Imports

Yogesh Sheoran
Hi, I am relatively new with pyinstaller and could not find any reference/help with my requirement.
I am creating an application which is using all the major ML frameworks like tensorflow, pytorch, onnx, caffe, mxnet. While packaging it with pyInstaller, I am able to pack it successfully but exe size is around 1.5GB.
I was wondering is there any way to tell/point the python interpretor packed with pyinstaller to look for these imports on the host machine?
There is one option in pyinstaller "exclude" but this excluded module can not import something which is not already imported by other packed modules.

--
You received this message because you are subscribed to the Google Groups "PyInstaller" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/pyinstaller/c5b4c9b0-ff9f-47f5-9cc0-837808822abfn%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

RE: External Imports

Steve Barnes

I suspect that you are using an Anaconda installation which includes just about everything (alternatively you are not working in a minimal venv) - try:

  1. Install Python from https://python.org
  2. Create and activate a virtual machine using python -mvenv
  3. pip install pysintaller and just the libraries that you actually use
  4. test to make sure everything is working
  5. run pyinstaller

 

You should end up with a much smaller package.

 

Steve Barnes

 

From: [hidden email] <[hidden email]> On Behalf Of Yogesh Sheoran
Sent: 04 May 2021 06:22
To: PyInstaller <[hidden email]>
Subject: [PyInstaller] External Imports

 

Hi, I am relatively new with pyinstaller and could not find any reference/help with my requirement.

I am creating an application which is using all the major ML frameworks like tensorflow, pytorch, onnx, caffe, mxnet. While packaging it with pyInstaller, I am able to pack it successfully but exe size is around 1.5GB.

I was wondering is there any way to tell/point the python interpretor packed with pyinstaller to look for these imports on the host machine?

There is one option in pyinstaller "exclude" but this excluded module can not import something which is not already imported by other packed modules.

 

--
You received this message because you are subscribed to the Google Groups "PyInstaller" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/pyinstaller/c5b4c9b0-ff9f-47f5-9cc0-837808822abfn%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "PyInstaller" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/pyinstaller/VI1PR03MB44799649BB6F09E0978AAF819B5A9%40VI1PR03MB4479.eurprd03.prod.outlook.com.
Reply | Threaded
Open this post in threaded view
|

Re: External Imports

bwoodsend
You don't specify if you are using onefile (which compresses it) or not so I don't know if this is 1.5GB raw or after compression. Recent tensorflow versions genuinely are about 1.5GB in size uncompressed so if you're using onedir then that is it I'm afraid. The only way around that is to downgrade to an older tensorflow when they were only ~200MB. If you're already using onefile (which, despite reducing the size, is usually is a bad idea for huge binaries because it takes ages to unpack) then it is probably as Steve says - other rubbish is getting dragged in.

On Tuesday, May 4, 2021 at 8:31:02 AM UTC+1 [hidden email] wrote:

I suspect that you are using an Anaconda installation which includes just about everything (alternatively you are not working in a minimal venv) - try:

  1. Install Python from https://python.org
  2. Create and activate a virtual machine using python -mvenv
  3. pip install pysintaller and just the libraries that you actually use
  4. test to make sure everything is working
  5. run pyinstaller

 

You should end up with a much smaller package.

 

Steve Barnes

 

From: [hidden email] <[hidden email]> On Behalf Of Yogesh Sheoran
Sent: 04 May 2021 06:22
To: PyInstaller <[hidden email]>
Subject: [PyInstaller] External Imports

 

Hi, I am relatively new with pyinstaller and could not find any reference/help with my requirement.

I am creating an application which is using all the major ML frameworks like tensorflow, pytorch, onnx, caffe, mxnet. While packaging it with pyInstaller, I am able to pack it successfully but exe size is around 1.5GB.

I was wondering is there any way to tell/point the python interpretor packed with pyinstaller to look for these imports on the host machine?

There is one option in pyinstaller "exclude" but this excluded module can not import something which is not already imported by other packed modules.

 

--
You received this message because you are subscribed to the Google Groups "PyInstaller" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/pyinstaller/c5b4c9b0-ff9f-47f5-9cc0-837808822abfn%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "PyInstaller" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/pyinstaller/608e78af-47e2-40f4-b64c-8c054bec7e37n%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: External Imports

Yogesh Sheoran
In reply to this post by Steve Barnes
Sorry, I think I did not asked clearly.
I am taking care of this thing, only required packages are getting packed that is not the issue for me.
What I need help in is - Not pack these packages at all, I am providing a installation script which will setup a virtual env and install the list of requirements(heavy packages). How can I make my exe(the embedded python interpreter) to use those packages?
On Tuesday, 4 May 2021 at 13:01:02 UTC+5:30 [hidden email] wrote:

I suspect that you are using an Anaconda installation which includes just about everything (alternatively you are not working in a minimal venv) - try:

  1. Install Python from https://python.org
  2. Create and activate a virtual machine using python -mvenv
  3. pip install pysintaller and just the libraries that you actually use
  4. test to make sure everything is working
  5. run pyinstaller

 

You should end up with a much smaller package.

 

Steve Barnes

 

From: [hidden email] <[hidden email]> On Behalf Of Yogesh Sheoran
Sent: 04 May 2021 06:22
To: PyInstaller <[hidden email]>
Subject: [PyInstaller] External Imports

 

Hi, I am relatively new with pyinstaller and could not find any reference/help with my requirement.

I am creating an application which is using all the major ML frameworks like tensorflow, pytorch, onnx, caffe, mxnet. While packaging it with pyInstaller, I am able to pack it successfully but exe size is around 1.5GB.

I was wondering is there any way to tell/point the python interpretor packed with pyinstaller to look for these imports on the host machine?

There is one option in pyinstaller "exclude" but this excluded module can not import something which is not already imported by other packed modules.

 

--
You received this message because you are subscribed to the Google Groups "PyInstaller" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/pyinstaller/c5b4c9b0-ff9f-47f5-9cc0-837808822abfn%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "PyInstaller" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/pyinstaller/458601f9-96e9-4885-a3a7-d46f914c829dn%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

RE: External Imports

Steve Barnes

Got you - your top level script will need to modify sys.path so as to include the location(s) where your virtual environment will install the installed packages in the list of places that it searches for the packages - normally I believe pysinstaller sets this to the exe files containing directory plus a zip file of the packages that pyinstaller has stored the included packages. Obviously enough this will need to be done prior to importing any such packages. - Personally I would prepend the location using sys.path.insert(0, wherever) so that the pip installed packages are used before any included ones.

 

Steve

 

From: [hidden email] <[hidden email]> On Behalf Of Yogesh Sheoran
Sent: 05 May 2021 09:29
To: PyInstaller <[hidden email]>
Subject: Re: [PyInstaller] External Imports

 

Sorry, I think I did not asked clearly.
I am taking care of this thing, only required packages are getting packed that is not the issue for me.

What I need help in is - Not pack these packages at all, I am providing a installation script which will setup a virtual env and install the list of requirements(heavy packages). How can I make my exe(the embedded python interpreter) to use those packages?

On Tuesday, 4 May 2021 at 13:01:02 UTC+5:30 [hidden email] wrote:

I suspect that you are using an Anaconda installation which includes just about everything (alternatively you are not working in a minimal venv) - try:

  1. Install Python from https://python.org
  2. Create and activate a virtual machine using python -mvenv
  3. pip install pysintaller and just the libraries that you actually use
  4. test to make sure everything is working
  5. run pyinstaller

 

You should end up with a much smaller package.

 

Steve Barnes

 

From: [hidden email] <[hidden email]> On Behalf Of Yogesh Sheoran
Sent: 04 May 2021 06:22
To: PyInstaller <[hidden email]>
Subject: [PyInstaller] External Imports

 

Hi, I am relatively new with pyinstaller and could not find any reference/help with my requirement.

I am creating an application which is using all the major ML frameworks like tensorflow, pytorch, onnx, caffe, mxnet. While packaging it with pyInstaller, I am able to pack it successfully but exe size is around 1.5GB.

I was wondering is there any way to tell/point the python interpretor packed with pyinstaller to look for these imports on the host machine?

There is one option in pyinstaller "exclude" but this excluded module can not import something which is not already imported by other packed modules.

 

--
You received this message because you are subscribed to the Google Groups "PyInstaller" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/pyinstaller/c5b4c9b0-ff9f-47f5-9cc0-837808822abfn%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "PyInstaller" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/pyinstaller/458601f9-96e9-4885-a3a7-d46f914c829dn%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "PyInstaller" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/pyinstaller/VI1PR03MB4479ABCCF5CCC77227A6A8499B589%40VI1PR03MB4479.eurprd03.prod.outlook.com.