## Description
This PR replaces all the `Playground::setup` calls with the new injected
`Playground`.
Most of the changes are done via [this
script](https://tangled.org/cptpiepmatz.de/nushell_playground_asr) which
uses the rust-analyzer internals to replace code via the AST.
All the test should behave exactly the same as before but they don't
need to manually set up a playground anymore. Also this removes quite a
lot of lines, which is nice.
## User-facing changes (Release notes)
n/a
---
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
## Description
This continues the AST → IR call migration (#18808, #18813, #18967). One
commit per step:
1. **Drop argument AST retention.** No command has opted into
`requires_ast_for_arguments` since #18813, so every `IrBlock.ast` entry
was `None`. The opt-in, `IrAstRef`, `IrBlock.ast`, the `ast` field of
`engine::Argument` and `engine::Call::positional_nth` are gone.
`Let::run`, which only an indirect call can reach, now returns an error
like `unlet`'s.
2. **One IR path for running a command by id.**
`nu_engine::eval_ir_call` runs an `ir::Call` the way compiled code does:
input type check, argument normalization, and parameter binding for a
custom command. The VM's call instruction shares the same code. `open`,
`http`, `save`, `%(...)` and a plugin's `call_decl` use it, which fixes
custom commands in three of them.
3. **Display call sites.** `help` (`nu-highlight`, `table`, `ansi`),
`tutor`, `print`, the DAP `print`, `clip copy` and table rendering for
an external's stdin build IR calls. `tutor` highlights through `help`'s
`nu_highlight_string`, so a snippet it can't highlight shows as plain
text instead of disappearing. `engine::Call::new` is deleted.
`CallImpl::to_owned` still turns an `AstRef` into an `AstBox`; phase 4
removes both variants.
## User-facing changes (Release notes)
- `http` commands can convert a response with a custom `from <ext>`
command, as `open` already could.
- `save` binds the flags and defaults of a custom `to <ext>` command.
- A plugin's `call_decl` can call a custom command.
## Breaking changes
### Scripts
- `save` checks a custom `to <ext>` converter's input type, as a direct
call does. A converter whose signature doesn't cover the data now errors
instead of running. Built-in `to` commands are unaffected: inputs they
already rejected now fail with `Input type not supported` instead of the
converter's own message. `open` and `http` hand converters an untyped
byte stream, which any input type accepts, so they don't change.
- `save` keeps a custom `to <ext>` converter's `$env` changes inside it
unless it is `def --env`, the same as `... | to <ext>`.
- `%($cmd)` checks its input and arguments as a direct call does.
Spreading a list into a command without a rest parameter is an error
instead of being ignored.
- `view ir --json` no longer has an `ast` field in `ir_block`. It was
always `null`.
| Command | main | branch |
|---|---|---|
| `def "to foo" []: int -> string { "int" }; [1 2] \| save out.foo; open
--raw out.foo` | `int` | `Input type not supported` (`only int input
data is supported`, input type `list<int>`) |
| `def "to foo" [] { $env.FOO = 1; "x" }; [1] \| save out.foo; $env.FOO?
\| describe` | `int` | `nothing` |
| `def --env "to foo" [] { $env.FOO = 1; "x" }; [1] \| save out.foo;
$env.FOO? \| describe` | `int` | `int` |
| `let cmd = 'length'; let args = [1]; [1 2] \| %($cmd) ...$args` | `2`
(the spread is dropped) | `Cannot spread a list into this command`
(`length ...$args` is a parse error) |
| `view ir --json {\|\| 1 } \| from json \| get ir_block \| columns` |
`[instructions, spans, data, ast, comments, register_count, file_count]`
| `[instructions, spans, data, comments, register_count, file_count]` |
### Plugins
`call_decl` checks the input type, including empty input, as a direct
call does. A call that the command's signature doesn't allow now errors,
the same as when the call is written in a script.
| Command (with `nu_plugin_example`) | main | branch |
|---|---|---|
| `'a' \| example call-decl uniq` | `[a]` | `Input type not supported`
(`only list<any> input data is supported`) |
| `example call-decl 'str join'` | `""` | `Pipeline empty` |
The `IrBlock` returned by the `get_block_ir` engine call no longer has
an `ast` field. Nushell only loads plugins built for the same minor
version, so a plugin never reads the old format from a newer engine.
### Rust API (embedders and out-of-tree `Command` implementations)
Plugins implement `PluginCommand` and are not affected.
| Removed | Instead |
|---|---|
| `engine::Call::new` | Build an `ir::Call` and run it with
`nu_engine::eval_ir_call` (example below). |
| `Command::requires_ast_for_arguments` | Delete the override. No
command has needed it since #18813. |
| `engine::Call::positional_nth`, `ir::Call::positional_ast`,
`engine::Argument::ast_expression` | Read argument values with `CallExt`
(`req`, `opt`, `rest`). On a compiled call these always returned `None`.
|
| The `ast` field of `engine::Argument::{Positional, Spread, Named}` |
Drop it from patterns and struct literals. |
| `ir::IrAstRef`, `IrBlock.ast` | Nothing. |
```diff
-let call = engine::Call::new(span);
-decl.run(engine_state, stack, &call, input)
+let call = ir::Call::build(decl_id, span).finish();
+nu_engine::eval_ir_call(engine_state, stack, &call, input)
```
With arguments, add them to the builder and call `leave` afterwards:
```rust
let mut builder = ir::Call::build(decl_id, span);
builder.add_named(stack, "width", "", span, Value::int(80, span));
let call = builder.finish();
let result = nu_engine::eval_ir_call(engine_state, stack, &call, input);
call.leave(stack);
```
Unlike `decl.run`, `eval_ir_call` also runs custom commands, checks the
input type and normalizes the arguments. Code that can't depend on
`nu-engine` can still pass `&(&call).into()` to `decl.run`.
`ir::Call::leave` on a call without arguments now does nothing. It used
to clear the whole argument stack, including the caller's arguments.
## Additional notes
- Closes#16974.
- Added `nu_engine::eval_ir_call`, which returns an error unless the
call's arguments are the last ones on the stack, and made
`nu_engine::nu_highlight_string` public.
- Next: phase 4 deletes `CallImpl::AstRef`/`AstBox` and the AST
evaluator's call path.
## Description
Changes the way that custom types are exposed in order to allow them to
be used in type signatures. This also enabled them to be resolved
correctly in the LSP.
## User-facing changes (Release notes)
- Custom types can now be used within type signatures. For plugins,
custom types for all installed plugins can be use. They will resolve
appropriately in the Nushell LSP. The following example show how to set
the return type for a command
```
def f [v: semver]: semver -> semver { $v } │
def ls-df [ ]: nothing -> polars_dataframe {
ls | polars into-df
}
```
## Description
Added a new `value_types` module under `nu_test_support` for helper
types (implementing `FromValue`) to work with output values in tests.
Currently contains a single type, `CompleteResult` for the output of the
`complete` command. Makes it easier to work with external processes in
tests.
## User-facing changes (Release notes)
N/A
<!--
Thank you for improving Nushell!
Please, read our contributing guide:
https://github.com/nushell/nushell/blob/main/CONTRIBUTING.md
-->
This PR enables the clippy lint `needless_raw_strings` to avoid raw
strings where they aren't needed. We have a lot of them and I don't
think that's that nice tbh.
## Release notes summary - What our users need to know
<!--
This section will be included as part of our release notes. See the
contributing guide for more details.
Please include only details relevant for users here. Motivation and
technical details can be added above or below this section.
You may leave this section blank until your PR is finalized. Ask a core
team member if you need help filling this section.
-->
n/a
<!--
Thank you for improving Nushell!
Please, read our contributing guide:
https://github.com/nushell/nushell/blob/main/CONTRIBUTING.md
-->
Closes#17569
Related to #12477
## Description
Switched `from ini` from `Ini::load_from_str` to
`Ini::load_from_str_opt` and wired command flags to rust-ini
`ParseOption` fields.
Added flags:
- `--no-quote`
- `--no-escape`
- `--indented-multiline-value`
- `--preserve-key-leading-whitespace`
Default behavior is unchanged when flags are not provided.
## Tests + formatting checks
- Ran `target/debug/nu -c "use toolkit.nu; toolkit check pr"` (passes)
- Validated on Windows: `from ini --no-escape` correctly parses paths
like `C:\Windows\System32\xcopy.exe`
## Release notes summary - What our users need to know
<!--
This section will be included as part of our release notes. See the
contributing guide for more details.
Please include only details relevant for users here. Motivation and
technical details can be added above or below this section.
You may leave this section blank until your PR is finalized. Ask a core
team member if you need help filling this section.
-->
`from ini` now supports rust-ini ParseOption flags (`--no-quote`,
`--no-escape`, `--indented-multiline-value`,
`--preserve-key-leading-whitespace`) for better compatibility with INI
variations.
In particular, `from ini --no-escape` allows Windows-style paths with
backslashes (including `\x`) to be parsed literally.
## Tasks after submitting
<!-- Remove any tasks which aren't relevant for your PR, or add your own
-->
- [ ] Update the
[documentation](https://github.com/nushell/nushell.github.io)
Been wanting to do this for a while and found some time. This PR
refactors all the command line arguments and parsing. The intent here is
to clean it up and use a separate cli arg parser. I chose lexopt because
it's tiny and seems to work vs something like clap which is just too
heavy.
Here are some features of this refactor. Some of these already existed
prior to this PR.
* better organized --help separated into sections (see screenshot below)
* cli args have descriptions in help display
* cli args have examples in help display
* cli args support "Do you mean?"
* cli args show available options when nothing is provided like `nu -m`
(table modes)
* switches can be combined like `nu -abc` meaning `nu -a -b -c`
* params with values work like this `nu -c foo` or `nu -c=foo`
* params with lists can use nushell lists in quotes e.g. `nu
--experimental-options "[example=false, reorder-cell-paths=true,
pipefail=true, enforce-runtime-annotations=true]"` or `nu --log-include
[warn, debug]`
* there are many tests (94 if i counted right) that try to protect the
cli args so that they stay working
<img width="1920" height="1040" alt="image"
src="https://github.com/user-attachments/assets/8335064a-469d-49da-8969-a12ff21bc5d8"
/>
<img width="1920" height="1009" alt="image"
src="https://github.com/user-attachments/assets/6eef7afb-286f-4b8e-9b55-f3acd149a037"
/>
<img width="1920" height="232" alt="image"
src="https://github.com/user-attachments/assets/be640070-6655-4d3d-981d-90a64da47a4b"
/>
## Release notes summary - What our users need to know
Refactor cli arg passing to make it more robust and testable
## Tasks after submitting
- [x] make sure tests pass on windows.
- [x] make sure tests pass on linux
- [x] make sure tests pass on macos
- fixes#16011
# Description
`Display` implementation for `f64` omits the decimal part for round
numbers, and by using it we did the same.
This affected:
- conversions to delimited formats: `csv`, `tsv`
- textual formats: `html`, `md`, `text`
- pretty printed `json` (`--raw` was unaffected)
- how single float values are displayed in the REPL
> [!TIP]
> This PR fixes our existing json pretty printing implementation.
> We can likely switch to using serde_json's impl using its
PrettyFormatter which allows arbitrary indent strings.
# User-Facing Changes
- Round trips through `csv`, `tsv`, and `json` preserve the type of
round floats.
- It's always clear whether a number is an integer or a float in the
REPL
```nushell
4 / 2
# => 2 # before: is this an int or a float?
4 / 2
# => 2.0 # after: clearly a float
```
# Tests + Formatting
Adjusted tests for the new behavior.
- 🟢 toolkit fmt
- 🟢 toolkit clippy
- 🟢 toolkit test
- 🟢 toolkit test stdlib
# After Submitting
N/A
---------
Co-authored-by: Bahex <17417311+Bahex@users.noreply.github.com>
# Description
Fixes#9942
This adds a new `--minified` flag to `to nuon` which removes all
possible white space. I added an example test to demonstrate the
functionality.
# User-Facing Changes
New flag becomes available to the user.
As I'm working on PWD-per-drive feature. Once the plugin test of env
failed. I checked the log, found sometime Windows can give drive letter
as lowercase, so the test case should be rewrite to check first letter
caseinsensitive equal, and following part normal equal.
```
assert_eq! failed at tests/plugins/env.rs:43:5
left: r"e:\Study\Nushell"
right: r"E:\Study\Nushell"
```
---------
Co-authored-by: Zhenping Zhao <pegasus.cadence@gmail.com>
# Description
[Context on
Discord](https://discord.com/channels/601130461678272522/855947301380947968/1292279795035668583)
**This is a breaking change, due to the removal of `is_running`.**
Some users find the `plugin list` command confusing, because it doesn't
show anything different after running `plugin add` or `plugin rm`. This
modifies the `plugin list` command to also look at the plugin registry
file to give some idea of how the plugins in engine state differ from
those in the plugin registry file.
The following values of `status` are now produced instead of
`is_running`:
- `added`: The plugin is present in the plugin registry file, but not in
the engine.
- `loaded`: The plugin is present both in the plugin registry file and
in the engine, but is not running.
- `running`: The plugin is currently running, and the `pid` column
should contain its process ID.
- `modified`: The plugin state present in the plugin registry file is
different from the state in the engine.
- `removed`: The plugin is still loaded in the engine, but is not
present in the plugin registry file.
- `invalid`: The data in the plugin registry file couldn't be
deserialized, and the plugin most likely needs to be added again.
Example (`commands` omitted):
```
╭──────┬─────────────────────┬────────────┬───────────┬──────────┬─────────────────────────────────────────────────────┬─────────╮
│ # │ name │ version │ status │ pid │ filename │ shell │
├──────┼─────────────────────┼────────────┼───────────┼──────────┼─────────────────────────────────────────────────────┼─────────┤
│ 0 │ custom_values │ 0.1.0 │ loaded │ │ /home/devyn/.cargo/bin/nu_plugin_custom_values │ │
│ 1 │ dbus │ 0.11.0 │ loaded │ │ /home/devyn/.cargo/bin/nu_plugin_dbus │ │
│ 2 │ example │ 0.98.1 │ loaded │ │ /home/devyn/.cargo/bin/nu_plugin_example │ │
│ 3 │ explore_ir │ 0.3.0 │ loaded │ │ /home/devyn/.cargo/bin/nu_plugin_explore_ir │ │
│ 4 │ formats │ 0.98.1 │ loaded │ │ /home/devyn/.cargo/bin/nu_plugin_formats │ │
│ 5 │ gstat │ 0.98.1 │ running │ 236662 │ /home/devyn/.cargo/bin/nu_plugin_gstat │ │
│ 6 │ inc │ 0.98.1 │ loaded │ │ /home/devyn/.cargo/bin/nu_plugin_inc │ │
│ 7 │ polars │ 0.98.1 │ added │ │ /home/devyn/.cargo/bin/nu_plugin_polars │ │
│ 8 │ query │ 0.98.1 │ removed │ │ /home/devyn/.cargo/bin/nu_plugin_query │ │
│ 9 │ stress_internals │ 0.98.1 │ loaded │ │ /home/devyn/.cargo/bin/nu_plugin_stress_internals │ │
╰──────┴─────────────────────┴────────────┴───────────┴──────────┴─────────────────────────────────────────────────────┴─────────╯
```
# User-Facing Changes
To `plugin list`:
* **Breaking:** The `is_running` column is removed and replaced with
`status`. Use `status == running` to filter equivalently.
* The `--plugin-config` from other plugin management commands is now
supported.
* Added an `--engine` flag which behaves more or less like before, and
doesn't load the plugin registry file at all.
* Added a `--registry` flag which only checks the plugin registry file.
All plugins appear as `added` since there is no state to compare with.
Because the default is to check both, the `plugin list` command might be
a little bit slower. If you don't need to check the plugin registry
file, the `--engine` flag does not load the plugin registry file at all,
so it should be just as fast as before.
# Tests + Formatting
Added tests for `added` and `removed` statuses. `modified` and `invalid`
are a bit more tricky so I didn't try.
# After Submitting
- [ ] update documentation that references the `plugin list` command
- [ ] release notes
# Description
[Context on
Discord](https://discord.com/channels/601130461678272522/855947301380947968/1292279795035668583)
**This is a breaking change, due to the removal of `is_running`.**
Some users find the `plugin list` command confusing, because it doesn't
show anything different after running `plugin add` or `plugin rm`. This
modifies the `plugin list` command to also look at the plugin registry
file to give some idea of how the plugins in engine state differ from
those in the plugin registry file.
The following values of `status` are now produced instead of
`is_running`:
- `added`: The plugin is present in the plugin registry file, but not in
the engine.
- `loaded`: The plugin is present both in the plugin registry file and
in the engine, but is not running.
- `running`: The plugin is currently running, and the `pid` column
should contain its process ID.
- `modified`: The plugin state present in the plugin registry file is
different from the state in the engine.
- `removed`: The plugin is still loaded in the engine, but is not
present in the plugin registry file.
- `invalid`: The data in the plugin registry file couldn't be
deserialized, and the plugin most likely needs to be added again.
Example (`commands` omitted):
```
╭──────┬─────────────────────┬────────────┬───────────┬──────────┬─────────────────────────────────────────────────────┬─────────╮
│ # │ name │ version │ status │ pid │ filename │ shell │
├──────┼─────────────────────┼────────────┼───────────┼──────────┼─────────────────────────────────────────────────────┼─────────┤
│ 0 │ custom_values │ 0.1.0 │ loaded │ │ /home/devyn/.cargo/bin/nu_plugin_custom_values │ │
│ 1 │ dbus │ 0.11.0 │ loaded │ │ /home/devyn/.cargo/bin/nu_plugin_dbus │ │
│ 2 │ example │ 0.98.1 │ loaded │ │ /home/devyn/.cargo/bin/nu_plugin_example │ │
│ 3 │ explore_ir │ 0.3.0 │ loaded │ │ /home/devyn/.cargo/bin/nu_plugin_explore_ir │ │
│ 4 │ formats │ 0.98.1 │ loaded │ │ /home/devyn/.cargo/bin/nu_plugin_formats │ │
│ 5 │ gstat │ 0.98.1 │ running │ 236662 │ /home/devyn/.cargo/bin/nu_plugin_gstat │ │
│ 6 │ inc │ 0.98.1 │ loaded │ │ /home/devyn/.cargo/bin/nu_plugin_inc │ │
│ 7 │ polars │ 0.98.1 │ added │ │ /home/devyn/.cargo/bin/nu_plugin_polars │ │
│ 8 │ query │ 0.98.1 │ removed │ │ /home/devyn/.cargo/bin/nu_plugin_query │ │
│ 9 │ stress_internals │ 0.98.1 │ loaded │ │ /home/devyn/.cargo/bin/nu_plugin_stress_internals │ │
╰──────┴─────────────────────┴────────────┴───────────┴──────────┴─────────────────────────────────────────────────────┴─────────╯
```
# User-Facing Changes
To `plugin list`:
* **Breaking:** The `is_running` column is removed and replaced with
`status`. Use `status == running` to filter equivalently.
* The `--plugin-config` from other plugin management commands is now
supported.
* Added an `--engine` flag which behaves more or less like before, and
doesn't load the plugin registry file at all.
* Added a `--registry` flag which only checks the plugin registry file.
All plugins appear as `added` since there is no state to compare with.
Because the default is to check both, the `plugin list` command might be
a little bit slower. If you don't need to check the plugin registry
file, the `--engine` flag does not load the plugin registry file at all,
so it should be just as fast as before.
# Tests + Formatting
Added tests for `added` and `removed` statuses. `modified` and `invalid`
are a bit more tricky so I didn't try.
# After Submitting
- [ ] update documentation that references the `plugin list` command
- [ ] release notes
# Description
The meaning of the word usage is specific to describing how a command
function is *used* and not a synonym for general description. Usage can
be used to describe the SYNOPSIS or EXAMPLES sections of a man page
where the permitted argument combinations are shown or example *uses*
are given.
Let's not confuse people and call it what it is a description.
Our `help` command already creates its own *Usage* section based on the
available arguments and doesn't refer to the description with usage.
# User-Facing Changes
`help commands` and `scope commands` will now use `description` or
`extra_description`
`usage`-> `description`
`extra_usage` -> `extra_description`
Breaking change in the plugin protocol:
In the signature record communicated with the engine.
`usage`-> `description`
`extra_usage` -> `extra_description`
The same rename also takes place for the methods on
`SimplePluginCommand` and `PluginCommand`
# Tests + Formatting
- Updated plugin protocol specific changes
# After Submitting
- [ ] update plugin protocol doc
# Description
This makes assignment operations and `const` behave the same way `let`
and `mut` do, absorbing the rest of the pipeline.
Changes the lexer to be able to recognize assignment operators as a
separate token, and then makes the lite parser continue to push spans
into the same command regardless of any redirections or pipes if an
assignment operator is encountered. Because the pipeline is no longer
split up by the lite parser at this point, it's trivial to just parse
the right hand side as if it were a subexpression not contained within
parentheses.
# User-Facing Changes
Big breaking change. These are all now possible:
```nushell
const path = 'a' | path join 'b'
mut x = 2
$x = random int
$x = [1 2 3] | math sum
$env.FOO = random chars
```
In the past, these would have led to (an attempt at) bare word string
parsing. So while `$env.FOO = bar` would have previously set the
environment variable `FOO` to the string `"bar"`, it now tries to run
the command named `bar`, hence the major breaking change.
However, this is desirable because it is very consistent - if you see
the `=`, you can just assume it absorbs everything else to the right of
it.
# Tests + Formatting
Added tests for the new behaviour. Adjusted some existing tests that
depended on the right hand side of assignments being parsed as
barewords.
# After Submitting
- [ ] release notes (breaking change!)
# Description
Adds functionality to the plugin interface to support calling internal
commands from plugins. For example, using `view ir --json`:
```rust
let closure: Value = call.req(0)?;
let Some(decl_id) = engine.find_decl("view ir")? else {
return Err(LabeledError::new("`view ir` not found"));
};
let ir_json = engine.call_decl(
decl_id,
EvaluatedCall::new(call.head)
.with_named("json".into_spanned(call.head), Value::bool(true, call.head))
.with_positional(closure),
PipelineData::Empty,
true,
false,
)?.into_value()?.into_string()?;
let ir = serde_json::from_value(&ir_json);
// ...
```
# User-Facing Changes
Plugin developers can now use `EngineInterface::find_decl()` and
`call_decl()` to call internal commands, which could be handy for
formatters like `to csv` or `to nuon`, or for reflection commands that
help gain insight into the engine.
# Tests + Formatting
- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`
# After Submitting
- [ ] release notes
- [ ] update plugin protocol documentation: `FindDecl`, `CallDecl`
engine calls; `Identifier` engine call response
<!--
if this PR closes one or more issues, you can automatically link the PR
with
them by using one of the [*linking
keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword),
e.g.
- this PR should close #xxxx
- fixes #xxxx
you can also mention related issues, PRs or discussions!
-->
# Description
<!--
Thank you for improving Nushell. Please, check our [contributing
guide](../CONTRIBUTING.md) and talk to the core team before making major
changes.
Description of your pull request goes here. **Provide examples and/or
screenshots** if your changes affect the user experience.
-->
The derive macros provided by #13031 are very useful for plugin authors.
In this PR I made use of these macros for two commands.
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the
tests for the standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
This Example usage could be highlighted in the changelog for plugin
authors as this is probably very useful for them.
# Description
This allows plugins to report their version (and potentially other
metadata in the future). The version is shown in `plugin list` and in
`version`.
The metadata is stored in the registry file, and reflects whatever was
retrieved on `plugin add`, not necessarily the running binary. This can
help you to diagnose if there's some kind of mismatch with what you
expect. We could potentially use this functionality to show a warning or
error if a plugin being run does not have the same version as what was
in the cache file, suggesting `plugin add` be run again, but I haven't
done that at this point.
It is optional, and it requires the plugin author to make some code
changes if they want to provide it, since I can't automatically
determine the version of the calling crate or anything tricky like that
to do it.
Example:
```
> plugin list | select name version is_running pid
╭───┬────────────────┬─────────┬────────────┬─────╮
│ # │ name │ version │ is_running │ pid │
├───┼────────────────┼─────────┼────────────┼─────┤
│ 0 │ example │ 0.93.1 │ false │ │
│ 1 │ gstat │ 0.93.1 │ false │ │
│ 2 │ inc │ 0.93.1 │ false │ │
│ 3 │ python_example │ 0.1.0 │ false │ │
╰───┴────────────────┴─────────┴────────────┴─────╯
```
cc @maxim-uvarov (he asked for it)
# User-Facing Changes
- `plugin list` gets a `version` column
- `version` shows plugin versions when available
- plugin authors *should* add `fn metadata()` to their `impl Plugin`,
but don't have to
# Tests + Formatting
Tested the low level stuff and also the `plugin list` column.
# After Submitting
- [ ] update plugin guide docs
- [ ] update plugin protocol docs (`Metadata` call & response)
- [ ] update plugin template (`fn metadata()` should be easy)
- [ ] release notes
# Description
This PR introduces a `ByteStream` type which is a `Read`-able stream of
bytes. Internally, it has an enum over three different byte stream
sources:
```rust
pub enum ByteStreamSource {
Read(Box<dyn Read + Send + 'static>),
File(File),
Child(ChildProcess),
}
```
This is in comparison to the current `RawStream` type, which is an
`Iterator<Item = Vec<u8>>` and has to allocate for each read chunk.
Currently, `PipelineData::ExternalStream` serves a weird dual role where
it is either external command output or a wrapper around `RawStream`.
`ByteStream` makes this distinction more clear (via `ByteStreamSource`)
and replaces `PipelineData::ExternalStream` in this PR:
```rust
pub enum PipelineData {
Empty,
Value(Value, Option<PipelineMetadata>),
ListStream(ListStream, Option<PipelineMetadata>),
ByteStream(ByteStream, Option<PipelineMetadata>),
}
```
The PR is relatively large, but a decent amount of it is just repetitive
changes.
This PR fixes#7017, fixes#10763, and fixes#12369.
This PR also improves performance when piping external commands. Nushell
should, in most cases, have competitive pipeline throughput compared to,
e.g., bash.
| Command | Before (MB/s) | After (MB/s) | Bash (MB/s) |
| -------------------------------------------------- | -------------:|
------------:| -----------:|
| `throughput \| rg 'x'` | 3059 | 3744 | 3739 |
| `throughput \| nu --testbin relay o> /dev/null` | 3508 | 8087 | 8136 |
# User-Facing Changes
- This is a breaking change for the plugin communication protocol,
because the `ExternalStreamInfo` was replaced with `ByteStreamInfo`.
Plugins now only have to deal with a single input stream, as opposed to
the previous three streams: stdout, stderr, and exit code.
- The output of `describe` has been changed for external/byte streams.
- Temporary breaking change: `bytes starts-with` no longer works with
byte streams. This is to keep the PR smaller, and `bytes ends-with`
already does not work on byte streams.
- If a process core dumped, then instead of having a `Value::Error` in
the `exit_code` column of the output returned from `complete`, it now is
a `Value::Int` with the negation of the signal number.
# After Submitting
- Update docs and book as necessary
- Release notes (e.g., plugin protocol changes)
- Adapt/convert commands to work with byte streams (high priority is
`str length`, `bytes starts-with`, and maybe `bytes ends-with`).
- Refactor the `tee` code, Devyn has already done some work on this.
---------
Co-authored-by: Devyn Cairns <devyn.cairns@gmail.com>
# Description
This PR adds a single test to assert interactivity on slow pipelines
Currently the timeout is set to 6 seconds, as the test can sometimes
take ~3secs to run on my local m1 mac air, which I don't think is an
indication of a slow pipeline, but rather slow test start up time...
# Description
Judiciously try to avoid allocations/clone by changing the signature of
functions
- **Don't pass str by value unnecessarily if only read**
- **Don't require a vec in `Sandbox::with_files`**
- **Remove unnecessary string clone**
- **Fixup unnecessary borrow**
- **Use `&str` in shape color instead**
- **Vec -> Slice**
- **Elide string clone**
- **Elide `Path` clone**
- **Take &str to elide clone in tests**
# User-Facing Changes
None
# Tests + Formatting
This touches many tests purely in changing from owned to borrowed/static
data
# Description
So far this seems like the winner of my poll on what the name should be.
I'll take this off draft once the poll expires, if this is indeed the
winner.
# Description
The local socket PR introduced a `Waitable` type, which could either
hold a value or be waited on until a value is available. Unlike a
channel, it would always return that value once set.
However, one issue with this design was that there was no way to detect
whether a value would ever be written. This splits the writer into a
different type `WaitableMut`, so that when it is dropped, waiting
threads can fail (because they'll never get a value).
# Tests + Formatting
A test has been added to `stress_internals` to make sure this fails in
the right way.
- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`
# Description
This allows the following commands to all accept a filename instead of a
plugin name:
- `plugin use`
- `plugin rm`
- `plugin stop`
Slightly complicated because of the need to also check against
`NU_PLUGIN_DIRS`, but I also fixed some issues with that at the same
time
Requested by @fdncred
# User-Facing Changes
The new commands are updated as described.
# Tests + Formatting
Tests for `NU_PLUGIN_DIRS` handling also made more robust.
- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`
# After Submitting
- [ ] Double check new docs to make sure they describe this capability
# Description
Adds a new keyword, `plugin use`. Unlike `register`, this merely loads
the signatures from the plugin cache file. The file is configurable with
the `--plugin-config` option either to `nu` or to `plugin use` itself,
just like the other `plugin` family of commands. At the REPL, one might
do this to replace `register`:
```nushell
> plugin add ~/.cargo/bin/nu_plugin_foo
> plugin use foo
```
This will not work in a script, because `plugin use` is a keyword and
`plugin add` does not evaluate at parse time (intentionally). This means
we no longer run random binaries during parse.
The `--plugins` option has been added to allow running `nu` with certain
plugins in one step. This is used especially for the `nu_with_plugins!`
test macro, but I'd imagine is generally useful. The only weird quirk is
that it has to be a list, and we don't really do this for any of our
other CLI args at the moment.
`register` now prints a deprecation parse warning.
This should fix#11923, as we now have a complete alternative to
`register`.
# User-Facing Changes
- Add `plugin use` command
- Deprecate `register`
- Add `--plugins` option to `nu` to replace a common use of `register`
# Tests + Formatting
I think I've tested it thoroughly enough and every existing test passes.
Testing nu CLI options and alternate config files is a little hairy and
I wish there were some more generic helpers for this, so this will go on
my TODO list for refactoring.
- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`
# After Submitting
- [ ] Update plugins sections of book
- [ ] Release notes
# Description
- Plugin signatures are now saved to `plugin.msgpackz`, which is
brotli-compressed MessagePack.
- The file is updated incrementally, rather than writing all plugin
commands in the engine every time.
- The file always contains the result of the `Signature` call to the
plugin, even if commands were removed.
- Invalid data for a particular plugin just causes an error to be
reported, but the rest of the plugins can still be parsed
# User-Facing Changes
- The plugin file has a different filename, and it's not a nushell
script.
- The default `plugin.nu` file will be automatically migrated the first
time, but not other plugin config files.
- We don't currently provide any utilities that could help edit this
file, beyond `plugin add` and `plugin rm`
- `from msgpackz`, `to msgpackz` could also help
- New commands: `plugin add`, `plugin rm`
# Tests + Formatting
Tests added for the format and for the invalid handling.
- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`
# After Submitting
- [ ] Check for documentation changes
- [ ] Definitely needs release notes
# Description
As suggested by @fdncred.
It's neat that this is possible, but the particularly useful part of
this is that we can actually
test it because it doesn't have any external dependencies, unlike the
python plugin.
Right now this just implements exactly the same behavior as the python
plugin, but we could have it
exercise a few more things.
Also fixes a couple of bugs:
- `.nu` plugins were not run with `nu --stdin`, so they couldn't take
input.
- `register` couldn't be called if `--no-config-file` was set, because
it would error on trying to
update the plugin file.
# User-Facing Changes
- `nu_plugin_nu_example` plugin added.
- `register` now works in `--no-config-file` mode.
# Tests + Formatting
Tests added for `nu_plugin_nu_example`.
- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`
# After Submitting
- [ ] Add the version bump to the release script just like for python
# Description
Adds support for running plugins using local socket communication
instead of stdio. This will be an optional thing that not all plugins
have to support.
This frees up stdio for use to make plugins that use stdio to create
terminal UIs, cc @amtoine, @fdncred.
This uses the [`interprocess`](https://crates.io/crates/interprocess)
crate (298 stars, MIT license, actively maintained), which seems to be
the best option for cross-platform local socket support in Rust. On
Windows, a local socket name is provided. On Unixes, it's a path. The
socket name is kept to a relatively small size because some operating
systems have pretty strict limits on the whole path (~100 chars), so on
macOS for example we prefer `/tmp/nu.{pid}.{hash64}.sock` where the hash
includes the plugin filename and timestamp to be unique enough.
This also adds an API for moving plugins in and out of the foreground
group, which is relevant for Unixes where direct terminal control
depends on that.
TODO:
- [x] Generate local socket path according to OS conventions
- [x] Add support for passing `--local-socket` to the plugin executable
instead of `--stdio`, and communicating over that instead
- [x] Test plugins that were broken, including
[amtoine/nu_plugin_explore](https://github.com/amtoine/nu_plugin_explore)
- [x] Automatically upgrade to using local sockets when supported,
falling back if it doesn't work, transparently to the user without any
visible error messages
- Added protocol feature: `LocalSocket`
- [x] Reset preferred mode to `None` on `register`
- [x] Allow plugins to detect whether they're running on a local socket
and can use stdio freely, so that TUI plugins can just produce an error
message otherwise
- Implemented via `EngineInterface::is_using_stdio()`
- [x] Clean up foreground state when plugin command exits on the engine
side too, not just whole plugin
- [x] Make sure tests for failure cases work as intended
- `nu_plugin_stress_internals` added
# User-Facing Changes
- TUI plugins work
- Non-Rust plugins could optionally choose to use this
- This might behave differently, so will need to test it carefully
across different operating systems
# Tests + Formatting
- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`
# After Submitting
- [ ] Document local socket option in plugin contrib docs
- [ ] Document how to do a terminal UI plugin in plugin contrib docs
- [ ] Document: `EnterForeground` engine call
- [ ] Document: `LeaveForeground` engine call
- [ ] Document: `LocalSocket` protocol feature
# Description
Because the plugin interface reader thread can be responsible for
sending a drop notification, it's possible for it to end up in a
deadlock where it's waiting for the response to the drop notification
call.
I decided that the best way to address this is to just discard the
response and not wait for it. It's not really important to synchronize
with the response to `Dropped`, so this is probably faster anyway.
cc @ayax79, this is your issue where polars is getting stuck
# User-Facing Changes
- A bug fix
- Custom value plugin: `custom-value handle update` command
# Tests + Formatting
Tried to add a test with a long pipeline with a lot of drops and run it
over and over to reproduce the deadlock.
- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`
# Description
This keeps plugin custom values that have requested drop notification
around during the lifetime of a plugin call / stream by sending them to
a channel that gets persisted during the lifetime of the call.
Before this change, it was very likely that the drop notification would
be sent before the plugin ever had a chance to handle the value it
received.
Tests have been added to make sure this works - see the `custom_values`
plugin.
cc @ayax79
# User-Facing Changes
This is basically just a bugfix, just a slightly big one.
However, I did add an `as_mut_any()` function for custom values, to
avoid having to clone them. This is a breaking change.
# Description
This is something that was discussed in the core team meeting last
Wednesday. @ayax79 is building `nu-plugin-polars` with all of the
dataframe commands into a plugin, and there are a lot of them, so it
would help to make the API more similar. At the same time, I think the
`Command` API is just better anyway. I don't think the difference is
justified, and the types for core commands have the benefit of requiring
less `.into()` because they often don't own their data
- Broke `signature()` up into `name()`, `usage()`, `extra_usage()`,
`search_terms()`, `examples()`
- `signature()` returns `nu_protocol::Signature`
- `examples()` returns `Vec<nu_protocol::Example>`
- `PluginSignature` and `PluginExample` no longer need to be used by
plugin developers
# User-Facing Changes
Breaking API for plugins yet again 😄
# Description
As suggested by @WindSoilder, since plugins can now contain both simple
commands that produce `Value` and commands that produce `PipelineData`
without having to choose one or the other for the whole plugin, this
change merges `stream_example` into `example`.
# User-Facing Changes
All of the example plugins are renamed.
# Tests + Formatting
- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`
# After Submitting
- [ ] Check nushell/nushell.github.io for any docs that match the
command names changed
Context: @abusch is working on a semver plugin with custom values and
wants users to be able to convert them back to strings
# Description
This allows `into string` to work on custom values if their base value
representation could be converted into a string with the same rules.
# User-Facing Changes
`into string` works on custom values.
Unfortunately, I couldn't really demo this with an example, because
there aren't any custom values that can be represented that way
included.
# Tests + Formatting
I was able to write a test using the custom values plugin.
- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`
# Description
@ayax79 says that the dataframe commands all have dataframe custom
values in their examples, and they're used for tests.
Rather than send the custom values to the engine, if they're in
examples, this change just renders them using `to_base_value()` first.
That way we avoid potentially having to hold onto custom values in
`plugins.nu` that might not be valid indefinitely - as will be the case
for dataframes in particular - but we still avoid forcing plugin writers
to not use custom values in their examples.
# User-Facing Changes
- Custom values usable in plugin examples
# Tests + Formatting
- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`
# After Submitting
# Description
Adds the `AddEnvVar` plugin call, which allows plugins to set
environment variables in the caller's scope. This is the first engine
call that mutates the caller's stack, and opens the door to more
operations like this if needed.
This also comes with an extra benefit: in doing this, I needed to
refactor how context was handled, and I was able to avoid cloning
`EngineInterface` / `Stack` / `Call` in most cases that plugin calls are
used. They now only need to be cloned if the plugin call returns a
stream. The performance increase is welcome (5.5x faster on `inc`!):
```nushell
# Before
> timeit { 1..100 | each { |i| $"2.0.($i)" | inc -p } }
405ms 941µs 952ns
# After
> timeit { 1..100 | each { |i| $"2.0.($i)" | inc -p } }
73ms 68µs 749ns
```
# User-Facing Changes
- New engine call: `add_env_var()`
- Performance enhancement for plugin calls
# Tests + Formatting
- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`
# After Submitting
- [x] Document env manipulation in plugins guide
- [x] Document `AddEnvVar` in plugin protocol
[Context on
Discord](https://discord.com/channels/601130461678272522/855947301380947968/1216517833312309419)
# Description
This is a significant breaking change to the plugin API, but one I think
is worthwhile. @ayax79 mentioned on Discord that while trying to start
on a dataframes plugin, he was a little disappointed that more wasn't
provided in terms of code organization for commands, particularly since
there are *a lot* of `dfr` commands.
This change treats plugins more like miniatures of the engine, with
dispatch of the command name being handled inherently, each command
being its own type, and each having their own signature within the trait
impl for the command type rather than having to find a way to centralize
it all into one `Vec`.
For the example plugins that have multiple commands, I definitely like
how this looks a lot better. This encourages doing code organization the
right way and feels very good.
For the plugins that have only one command, it's just a little bit more
boilerplate - but still worth it, in my opinion.
The `Box<dyn PluginCommand<Plugin = Self>>` type in `commands()` is a
little bit hairy, particularly for Rust beginners, but ultimately not so
bad, and it gives the desired flexibility for shared state for a whole
plugin + the individual commands.
# User-Facing Changes
Pretty big breaking change to plugin API, but probably one that's worth
making.
```rust
use nu_plugin::*;
use nu_protocol::{PluginSignature, PipelineData, Type, Value};
struct LowercasePlugin;
struct Lowercase;
// Plugins can now have multiple commands
impl PluginCommand for Lowercase {
type Plugin = LowercasePlugin;
// The signature lives with the command
fn signature(&self) -> PluginSignature {
PluginSignature::build("lowercase")
.usage("Convert each string in a stream to lowercase")
.input_output_type(Type::List(Type::String.into()), Type::List(Type::String.into()))
}
// We also provide SimplePluginCommand which operates on Value like before
fn run(
&self,
plugin: &LowercasePlugin,
engine: &EngineInterface,
call: &EvaluatedCall,
input: PipelineData,
) -> Result<PipelineData, LabeledError> {
let span = call.head;
Ok(input.map(move |value| {
value.as_str()
.map(|string| Value::string(string.to_lowercase(), span))
// Errors in a stream should be returned as values.
.unwrap_or_else(|err| Value::error(err, span))
}, None)?)
}
}
// Plugin now just has a list of commands, and the custom value op stuff still goes here
impl Plugin for LowercasePlugin {
fn commands(&self) -> Vec<Box<dyn PluginCommand<Plugin=Self>>> {
vec![Box::new(Lowercase)]
}
}
fn main() {
serve_plugin(&LowercasePlugin{}, MsgPackSerializer)
}
```
Time this however you like - we're already breaking stuff for 0.92, so
it might be good to do it now, but if it feels like a lot all at once,
it could wait.
# Tests + Formatting
- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`
# After Submitting
- [ ] Update examples in the book
- [x] Fix#12088 to match - this change would actually simplify it a
lot, because the methods are currently just duplicated between `Plugin`
and `StreamingPlugin`, but they only need to be on `Plugin` with this
change