Target release8.1.1
Epic
Document status
Document owner
Designer
Developers
QA


Goals

Enhanced auto-complete for Python 2, 3. This enhanced auto-complete must be able to:

This file can be used to test the behavior of this feature.

The completion features should rely on Jedi. Jedi is a python library that does static analysis of Python code to provide the relevant code completions. It is used by other project to provide code completion like the Python plugin for Emacs or VSCode. Using Jedi has many advantages for us:

The main downside is that we need to use a Python process to do the completion.

The development of this feature is currently happening on bitbucket under a jedi bookmark. This is an ongoing process. The feature can be tested on a 8.1 release of Netbeans with the nbms available here. The list of issues with the current nbms is available here.

In addition to the completion feature, the nbms provode code validation with pyflakes and linting with pep8. The reason behind this is that they work the same way (in a Python process) and will allow us to improve the support for python 3 (the current lint and error checking are not compatible with Python 3. For instance, print 3 is reported as valid python 3 code and print(3, file=stderr) is reported as invalid).

Background and strategic fit

  1. This is a competitive feature
  2. Very useful while programming and a must have for big projects
  3. Improve support for python 2 and python 3
  4. Less pains for the user: no need to use another IDE for python

Assumptions

Requirements

#TitleUser StoryImportanceNotes
1Basic supportMeet all requirement of enhanced completion defined earlier. No stackstrace, no apparent bugs.Must HaveDone
2ActivationJedi completion must be turned on by default within the languages option like for Java (Options > Editor > Code Completion then on the select list Python).Must Have

Currently, you must enable completion in this panel and then enable Jedi support in the misc panel.

Link to the proper code or documentation on how to achieve that would be really appreciated.

Done

3On the flyCurrently the user must use CTRL+SPC to activate completion. This must be available while typing like for Java or Javascript if the user choose to. This feature must be turned off by default like in Java and JavaScript.Must HaveDone
4Options for jediJedi supports a wide range of options. The user should be able to tune them.Good to haveVery likely to be pushed to a future release.
5Options for pep8The user should be able to tune the options of Pep8 globally, per project. The settings should be detected automatically if a project contains the "proper" configuration file. This will allow the user to use different rules than the default ones if he/she wishes.Good to have.Future release
6Options for pyfalkesThe user should be able to tune the options of pyflakes globally, per project. The settings should be detected automatically if a project contains the "proper" configuration file. This will allow the user to use different rules than the default ones if he/she wishes.Good to have.Future release
7Improve navigator support for Python 3 filesOn python 3 files with yield from navigator is empty. This should be improvedGood to HaveFuture release
8Use Jedi to complete function signaturesThis would allow the user to know what parameters are expected for a function. We should also do like in Java: set the focus on the first parameter, allow the user to enter it, once it is correct, the user presses the RETURN key and we switch to the edition of the first paramter.Good to Have

Future release

See: https://jedi.readthedocs.io/en/latest/docs/plugin-api.html#jedi.api.Script.call_signatures

9Use Jedi to provide navigation in source filesThe user CTRL+Click on a function/class/variable name and we take the user to the definition.Good to Have

Future release

See: https://jedi.readthedocs.io/en/latest/docs/plugin-api.html#jedi.api.Script.goto_definitions

s = jedi.api.Script(source='def t():\n    passprint(t())', line=3, column=6, path='/tm/test.py')

r = s.goto_definitions()[0]

r.line, r.column

10Use Jedi for refactoring

The user select a function/class/variable and rename it with a refactoring tool based or Script.usages to rename all the occurrences correctly.

Good to Have

Future release

See: https://jedi.readthedocs.io/en/latest/docs/plugin-api.html#jedi.api.Script.usages

User interaction and design

Add design for option panels.

Questions

Below is a list of questions to be addressed as a result of this requirements document:

QuestionOutcome
How do we make this compatible with Jython?One solution would be to ask the jedi developer to add support for Jython but the best way would be to report the missing package pydoc_data to the Jython project so they can implement it. That would benefit them since it will improve their compatibility with Python code. There's currently no mention of it in their bugtracker: http://bugs.jython.org/issue?%40columns=id%2Cactivity%2Ctitle%2Ccreator%2Cassignee%2Cstatus&%40sort=-activity&%40group=priority&%40filter=status&status=-1%2C1%2C3&%40search_text=pydoc_data&submit=search+in+open+issues.
or, how will CC work when Jython is used?Currently, the "standard" although incomplete completion still works. That's the only completion mechanism we have now. Maybe it can be improved.
How will Jedi be packaged?Currently Jedi is package in contrib/python.jedi, pyflakes in contrib/python.pyflakes and pep8 in python.pep8. The relevant python files are installed because of a patch in main. This is similar to how Jython is packaged.
Will packaging be dependent on Python version?Jedi, pyflakes and pep8 are design to run on any supported version of python (2.7 and >= 3.3). So we can package only one version of each software. If something doesn't work on a version of python, we should report the bug upstream and update our package.

Support on Jython

Jython is a python interpreter written in Java. Since it is not yet fully compatible with Python, some programs may not run. In this section, we detail how the auto-complete feature is impacted by this.

FeaturesSupported?Notes
PEP8YES
PyflakesPOSSIBLEWould require more testing. I (Jenselme) tested a month ago and I got an error but today (2016-04-18) I did a test with `./jython-2.7.0/bin/jython python_parse_process.py <<< '{"path": "python_parse_process.py", "source": "import sys\nprint 789\nprint zut\nprint(789, file=sys.stdout)\n"}'` an it seemed to work. However, if we use pyflakes classes, functions names and function parameters are not correctly highlighted. To do this highlighting, we rely on Jedi which doesn't work on Jython. I think that for now, we better rely on "standard" parsing with Jython.
JediNOJython doesn't implement pydoc_topics (python 2.7 and above) which is required for Jedi to work. See this feature request to fellow the implementation of this feature by jython.


Not Doing

Notes on the implementation

The completion services were inspired by their equivalent in nbts a NetBeans plugin for the TypeScript language by Everlaw. Many thanks to them for their work and inspiration.

The completion/syntax analysis features are provided by Python daemons running in the background. We have the following daemons:

This services are started if they are enabled in the options when opening a Python project or file. We launch one service per Python interpreter (the services are reused between projects/files with the same interpreter). We use the interpreter registered in the project properties to launch the services. Once all projects or files associated with an interpreter are closed, the services associated with this interpreter are stopped. The association between services, interpreters and projects/files is done in a static map in python.core/src/org/netbeans/modules/python/api/PythonProcess.java. Projects and files are registered with a call to PythonProcess.registerProject(projectDirectory.getPath(), platform.getInterpreterCommand()); and unregistered with a call to PythonProcess.unregisterProject(projectDirectory.getPath());. When the user changes the interpreter in the options, the mapping is updated and the proper services are stopped/started.

The Python scripts are designed not to crash (we catch all exceptions). They must always provide an answer (an empty one in case of a problem in the scripts) in order to prevent the IDE to hang.