Patchwork How to get back to a one-file `hg` executable?

Okay, patching up for main, I get this:

error[PYOXIDIZER_BUILD]: adding PythonModuleSource<name=hgdemandimport>

Caused by:
    0: adding PythonModuleSource<hgdemandimport>
    1: resource collector does not allow resources in filesystem-relative
  --> ./pyoxidizer.bzl:61:5
   |
61 |     exe.add_python_resources(exe.pip_install(["--verbose", ROOT]))
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
add_python_resources()


error: adding PythonModuleSource<name=hgdemandimport>

Caused by:
    0: adding PythonModuleSource<hgdemandimport>
    1: resource collector does not allow resources in filesystem-relative

so it's sad about hgdemandimport, but I'm not sure what to do with this
information. For the record, my patch to pyoxidizer.bzl is this:

     # We need this to make resourceutil happy, since it looks for
sys.frozen.


On Tue, Nov 3, 2020 at 9:54 AM Augie Fackler <augie@google.com> wrote:

> Sigh, you changed the config file again in an incompatible way
> (config.run_mode isn't a thing anymore) so I'll use the v0.9.0 tag. It
> still fails, but still not helpfully?
>
> Successfully installed mercurial-5.6-4-8711dc13474c-20201103
> Removed build tracker:
> '/private/var/folders/cy/2vk7k97j21qd6yr76gp6x5m8001p1v/T/pip-req-tracker-z19m5oxc'
> adding <PythonModuleSource>
> error[PYOXIDIZER_BUILD]: resource collector does not allow resources in
> filesystem-relative
>   --> ./pyoxidizer.bzl:63:9
>    |
> 63 |         exe.add_python_resource(r)
>    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^ add_python_resource
>
> error: resource collector does not allow resources in filesystem-relative
>
> I'll see if I can update the pyoxidizer.bzl in a bit here to work on main,
> I guess?
>
> On Mon, Nov 2, 2020 at 11:06 PM Gregory Szorc <gregory.szorc@gmail.com>
> wrote:
>
>> I'm unsure what's going on here.
>>
>> For this to occur, `.resources_location_fallback = None` must hold on the
>> policy instance and some resource being added must have its `.add_location
>> = "filesystem-relative:..."`.
>>
>> This error message is horrible. And I just committed a change to the
>> `main` branch to print more context about which exact resource is failing.
>> If you don't want to run the latest commit from `main`, you can change this
>> to something like:
>>
>> for resource in exe.pip_install["--verbose", ROOT]):
>>     print("adding %s" % resource)
>>     exe.add_python_resource(resource)
>>
>> Or you could look for references to "filesystem-relative" in the
>> configuration file. There's likely something setting that as the
>> `.add_location =` value.
>>
>> FWIW I _think_ Martin did all the necessary work to enable Mercurial to
>> be shipped as a standalone binary? I left it out of the in-tree config
>> because files mode was backwards compatible and safer. But a single file
>> binary is something we could implement next release.
>>
>> On Mon, Nov 2, 2020 at 11:06 AM Augie Fackler <augie@google.com> wrote:
>>
>>> Hi Greg!
>>>
>>> I'm working on updating hg and trying to switch us to pyoxidizer for
>>> Windows, but a speedbump on the way is that our Mac builds have broken:
>>> we'd been relying on the one-file build mode of the binary, and now that
>>> appears to be impossible? Am I missing something? I tried editing the
>>> pyoxidizer.bzl to make all resources be "in-memory" but then I get a
>>> failure trying to add the hg packages, eg:
>>>
>>> error[PYOXIDIZER_BUILD]: resource collector does not allow resources in
>>> filesystem-relative
>>>   --> ./pyoxidizer.bzl:64:5
>>>    |
>>> 64 |     exe.add_python_resources(exe.pip_install(["--verbose", ROOT]))
>>>    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>> add_python_resources()
>>>
>>> which left me stumped. Am I missing something obvious in the docs or
>>> configuration that would let us get back to one-file executables for our
>>> macOS users?
>>>
>>> Thanks,
>>> Augie
>>>
>>

Spent a little more time with the latest pyoxidizer, but started getting
weird results: complaining from pyoxidizer that it needed a .to_string() in
the run_command: Some("my string here") value, and if I patch around that
or comment out run_command I get a panic inside starlark on an optional,
which I don't think makes sense either.

I guess I need some guidance on how to debug why hgdemandimport thinks it
can't be loaded from the single-file executable? Any insight?

On Tue, Nov 3, 2020 at 11:11 AM Augie Fackler <augie@google.com> wrote:

> Okay, patching up for main, I get this:
>
> error[PYOXIDIZER_BUILD]: adding PythonModuleSource<name=hgdemandimport>
>
> Caused by:
>     0: adding PythonModuleSource<hgdemandimport>
>     1: resource collector does not allow resources in filesystem-relative
>   --> ./pyoxidizer.bzl:61:5
>    |
> 61 |     exe.add_python_resources(exe.pip_install(["--verbose", ROOT]))
>    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> add_python_resources()
>
>
> error: adding PythonModuleSource<name=hgdemandimport>
>
> Caused by:
>     0: adding PythonModuleSource<hgdemandimport>
>     1: resource collector does not allow resources in filesystem-relative
>
> so it's sad about hgdemandimport, but I'm not sure what to do with this
> information. For the record, my patch to pyoxidizer.bzl is this:
>
> diff --git a/rust/hgcli/pyoxidizer.bzl b/rust/hgcli/pyoxidizer.bzl
> --- a/rust/hgcli/pyoxidizer.bzl
> +++ b/rust/hgcli/pyoxidizer.bzl
> @@ -40,12 +40,11 @@ def make_exe(dist):
>      # extensions.
>      packaging_policy.extension_module_filter = "all"
>      packaging_policy.resources_location = "in-memory"
> -    packaging_policy.resources_location_fallback =
> "filesystem-relative:lib"
>      packaging_policy.register_resource_callback(resource_callback)
>
>      config = dist.make_python_interpreter_config()
>      config.raw_allocator = "system"
> -    config.run_mode = "eval:%s" % RUN_CODE
> +    config.run_command = RUN_CODE
>      # We want to let the user load extensions from the file system
>      config.filesystem_importer = True
>      # We need this to make resourceutil happy, since it looks for
> sys.frozen.
>
>
> On Tue, Nov 3, 2020 at 9:54 AM Augie Fackler <augie@google.com> wrote:
>
>> Sigh, you changed the config file again in an incompatible way
>> (config.run_mode isn't a thing anymore) so I'll use the v0.9.0 tag. It
>> still fails, but still not helpfully?
>>
>> Successfully installed mercurial-5.6-4-8711dc13474c-20201103
>> Removed build tracker:
>> '/private/var/folders/cy/2vk7k97j21qd6yr76gp6x5m8001p1v/T/pip-req-tracker-z19m5oxc'
>> adding <PythonModuleSource>
>> error[PYOXIDIZER_BUILD]: resource collector does not allow resources in
>> filesystem-relative
>>   --> ./pyoxidizer.bzl:63:9
>>    |
>> 63 |         exe.add_python_resource(r)
>>    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^ add_python_resource
>>
>> error: resource collector does not allow resources in filesystem-relative
>>
>> I'll see if I can update the pyoxidizer.bzl in a bit here to work on
>> main, I guess?
>>
>> On Mon, Nov 2, 2020 at 11:06 PM Gregory Szorc <gregory.szorc@gmail.com>
>> wrote:
>>
>>> I'm unsure what's going on here.
>>>
>>> For this to occur, `.resources_location_fallback = None` must hold on
>>> the policy instance and some resource being added must have its
>>> `.add_location = "filesystem-relative:..."`.
>>>
>>> This error message is horrible. And I just committed a change to the
>>> `main` branch to print more context about which exact resource is failing.
>>> If you don't want to run the latest commit from `main`, you can change this
>>> to something like:
>>>
>>> for resource in exe.pip_install["--verbose", ROOT]):
>>>     print("adding %s" % resource)
>>>     exe.add_python_resource(resource)
>>>
>>> Or you could look for references to "filesystem-relative" in the
>>> configuration file. There's likely something setting that as the
>>> `.add_location =` value.
>>>
>>> FWIW I _think_ Martin did all the necessary work to enable Mercurial to
>>> be shipped as a standalone binary? I left it out of the in-tree config
>>> because files mode was backwards compatible and safer. But a single file
>>> binary is something we could implement next release.
>>>
>>> On Mon, Nov 2, 2020 at 11:06 AM Augie Fackler <augie@google.com> wrote:
>>>
>>>> Hi Greg!
>>>>
>>>> I'm working on updating hg and trying to switch us to pyoxidizer for
>>>> Windows, but a speedbump on the way is that our Mac builds have broken:
>>>> we'd been relying on the one-file build mode of the binary, and now that
>>>> appears to be impossible? Am I missing something? I tried editing the
>>>> pyoxidizer.bzl to make all resources be "in-memory" but then I get a
>>>> failure trying to add the hg packages, eg:
>>>>
>>>> error[PYOXIDIZER_BUILD]: resource collector does not allow resources in
>>>> filesystem-relative
>>>>   --> ./pyoxidizer.bzl:64:5
>>>>    |
>>>> 64 |     exe.add_python_resources(exe.pip_install(["--verbose", ROOT]))
>>>>    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>>> add_python_resources()
>>>>
>>>> which left me stumped. Am I missing something obvious in the docs or
>>>> configuration that would let us get back to one-file executables for our
>>>> macOS users?
>>>>
>>>> Thanks,
>>>> Augie
>>>>
>>>

Closing the loop on this: Greg helped me out a bunch and we got this
working again. I'll send some patches next week to make pyoxidized `hg` be
one-file on Linux and macOS. :)

On Tue, Nov 3, 2020 at 6:41 PM Augie Fackler <augie@google.com> wrote:

> Spent a little more time with the latest pyoxidizer, but started getting
> weird results: complaining from pyoxidizer that it needed a .to_string() in
> the run_command: Some("my string here") value, and if I patch around that
> or comment out run_command I get a panic inside starlark on an optional,
> which I don't think makes sense either.
>
> I guess I need some guidance on how to debug why hgdemandimport thinks it
> can't be loaded from the single-file executable? Any insight?
>
> On Tue, Nov 3, 2020 at 11:11 AM Augie Fackler <augie@google.com> wrote:
>
>> Okay, patching up for main, I get this:
>>
>> error[PYOXIDIZER_BUILD]: adding PythonModuleSource<name=hgdemandimport>
>>
>> Caused by:
>>     0: adding PythonModuleSource<hgdemandimport>
>>     1: resource collector does not allow resources in filesystem-relative
>>   --> ./pyoxidizer.bzl:61:5
>>    |
>> 61 |     exe.add_python_resources(exe.pip_install(["--verbose", ROOT]))
>>    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>> add_python_resources()
>>
>>
>> error: adding PythonModuleSource<name=hgdemandimport>
>>
>> Caused by:
>>     0: adding PythonModuleSource<hgdemandimport>
>>     1: resource collector does not allow resources in filesystem-relative
>>
>> so it's sad about hgdemandimport, but I'm not sure what to do with this
>> information. For the record, my patch to pyoxidizer.bzl is this:
>>
>> diff --git a/rust/hgcli/pyoxidizer.bzl b/rust/hgcli/pyoxidizer.bzl
>> --- a/rust/hgcli/pyoxidizer.bzl
>> +++ b/rust/hgcli/pyoxidizer.bzl
>> @@ -40,12 +40,11 @@ def make_exe(dist):
>>      # extensions.
>>      packaging_policy.extension_module_filter = "all"
>>      packaging_policy.resources_location = "in-memory"
>> -    packaging_policy.resources_location_fallback =
>> "filesystem-relative:lib"
>>      packaging_policy.register_resource_callback(resource_callback)
>>
>>      config = dist.make_python_interpreter_config()
>>      config.raw_allocator = "system"
>> -    config.run_mode = "eval:%s" % RUN_CODE
>> +    config.run_command = RUN_CODE
>>      # We want to let the user load extensions from the file system
>>      config.filesystem_importer = True
>>      # We need this to make resourceutil happy, since it looks for
>> sys.frozen.
>>
>>
>> On Tue, Nov 3, 2020 at 9:54 AM Augie Fackler <augie@google.com> wrote:
>>
>>> Sigh, you changed the config file again in an incompatible way
>>> (config.run_mode isn't a thing anymore) so I'll use the v0.9.0 tag. It
>>> still fails, but still not helpfully?
>>>
>>> Successfully installed mercurial-5.6-4-8711dc13474c-20201103
>>> Removed build tracker:
>>> '/private/var/folders/cy/2vk7k97j21qd6yr76gp6x5m8001p1v/T/pip-req-tracker-z19m5oxc'
>>> adding <PythonModuleSource>
>>> error[PYOXIDIZER_BUILD]: resource collector does not allow resources in
>>> filesystem-relative
>>>   --> ./pyoxidizer.bzl:63:9
>>>    |
>>> 63 |         exe.add_python_resource(r)
>>>    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^ add_python_resource
>>>
>>> error: resource collector does not allow resources in filesystem-relative
>>>
>>> I'll see if I can update the pyoxidizer.bzl in a bit here to work on
>>> main, I guess?
>>>
>>> On Mon, Nov 2, 2020 at 11:06 PM Gregory Szorc <gregory.szorc@gmail.com>
>>> wrote:
>>>
>>>> I'm unsure what's going on here.
>>>>
>>>> For this to occur, `.resources_location_fallback = None` must hold on
>>>> the policy instance and some resource being added must have its
>>>> `.add_location = "filesystem-relative:..."`.
>>>>
>>>> This error message is horrible. And I just committed a change to the
>>>> `main` branch to print more context about which exact resource is failing.
>>>> If you don't want to run the latest commit from `main`, you can change this
>>>> to something like:
>>>>
>>>> for resource in exe.pip_install["--verbose", ROOT]):
>>>>     print("adding %s" % resource)
>>>>     exe.add_python_resource(resource)
>>>>
>>>> Or you could look for references to "filesystem-relative" in the
>>>> configuration file. There's likely something setting that as the
>>>> `.add_location =` value.
>>>>
>>>> FWIW I _think_ Martin did all the necessary work to enable Mercurial to
>>>> be shipped as a standalone binary? I left it out of the in-tree config
>>>> because files mode was backwards compatible and safer. But a single file
>>>> binary is something we could implement next release.
>>>>
>>>> On Mon, Nov 2, 2020 at 11:06 AM Augie Fackler <augie@google.com> wrote:
>>>>
>>>>> Hi Greg!
>>>>>
>>>>> I'm working on updating hg and trying to switch us to pyoxidizer for
>>>>> Windows, but a speedbump on the way is that our Mac builds have broken:
>>>>> we'd been relying on the one-file build mode of the binary, and now that
>>>>> appears to be impossible? Am I missing something? I tried editing the
>>>>> pyoxidizer.bzl to make all resources be "in-memory" but then I get a
>>>>> failure trying to add the hg packages, eg:
>>>>>
>>>>> error[PYOXIDIZER_BUILD]: resource collector does not allow resources
>>>>> in filesystem-relative
>>>>>   --> ./pyoxidizer.bzl:64:5
>>>>>    |
>>>>> 64 |     exe.add_python_resources(exe.pip_install(["--verbose", ROOT]))
>>>>>    |
>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>>>> add_python_resources()
>>>>>
>>>>> which left me stumped. Am I missing something obvious in the docs or
>>>>> configuration that would let us get back to one-file executables for our
>>>>> macOS users?
>>>>>
>>>>> Thanks,
>>>>> Augie
>>>>>
>>>>