API changes

GSF

GSF is a language infrastructure for NetBeans. The purpose of GSF is to help create first-class editing support for a group languages (like Ruby, Groovy, JavaScript, Python, Scala, PHP, etc). By "first-class" editing support I mean all the features that we currently have for Java in NetBeans. Deep editing support including intelligent code completion, quick fixes and hints, and so on.

The abbreviation "GSF" is an historical artifact and a new name should probably be assigned.

Philosophy

The GSF API is intended for "language plugins". These are modules which add support for new languages (such as Ruby, JavaScript, HTML, CSS and so on) to NetBeans. The key philosophy behind GSF is:

The language plugin deals only with language specific details.

The implication of this is that nearly all UI, and all "infrastructure" code like listening on editor changes, scheduling parsing, implementing editor actions etc. are handled by the GSF infrastructure. A language plugin should just focus on specific features as they pertain to this language.

If you are trying to get started writing a new language using GSF, see the Getting Started document, which will try to order the tasks involved and point to the relevant sections.

Example: Instant Rename

For an example of this, take a look at the following diagram. This shows how "Instant Rename" works in GSF. The Instant Rename action itself (editor action implementation, keybinding registration, etc.) is provided by GSF. The language plugin implements two services:

• A parser - which produces a parse tree. This is used for most features in GSF.
• An "InstantRenamer" - a simple interface where the language plugin is handed one of its own parse trees, and it needs to answer two questions:
  1. Is the given caret offset renameable? (And if not, provide some error message to explain why, or an indication that refactoring needs to be involved.
  )
  2. Produce a set of offset ranges (begin to end) for all the instances of this symbol that should be renamed together.

As you can see, GSF provides implementations for a lot of the UI and interactions that you need for deep editing support for your language. You implement your lexer, parser, and feature implementations like code completion, go to declaration, quickfixes etc. based on analyzing your own parse trees. You talk through feature specific APIs to GSF, and GSF does the rest.

Architecture

Here's a rough architecture diagram for GSF:


As you can see, language plugins talk mostly through the GSF APIs to implement their features. GSF is in turn implemented in many cases on top of existing NetBeans APIs. In some cases, the abstraction on top of existing APIs (such as the Hyperlink API) is very thin. In other cases, such as for quickfixes, mark occurrences or semantic highlighting, GSF provides a lot of UI implementation under the hood on top of the existing APIs. You might wonder why GSF should bother with features such as code completion and hyperlinking when the existing APIs are pretty complete. There are several reasons for this:

• Simpler to learn from one place. The goal for GSF is to bring language support of many languages in NetBeans up to the same depth of support as we have for Java. (See http://wiki.netbeans.org/EditorFeaturesChecklist for a document which tries to list all these features.) If you have to go and hunt for various different NetBeans APIs (navigation, hyperlinking, highlighting, etc.) it's a lot harder to make sure you cover everything, and you in some cases have to learn new concepts.
• GSF can take advantage of synergy. For example, GSF knows how to (a) parse your code, and (b) index your code, so it can hand you an up to date parse tree when it's asking you to for example compute the declaration location for an editor click. That makes your job easier. As another example, because GSF implements both your Hyperlink provider (Go To Declaration), and your code completion documentation provider, it can register a tooltip which shows the documentation for the symbol under the caret when you are ctrl-hovering over that symbol. And so on.
• GSF makes unit testing easier. This is another aspect of the synergy point I brought up above. When you for example add a new "semantic highlighting" implementation, writing a unit test for it is as simple as writing a simple function call - pass in a source file, and a caret location. The GSF unit testing infrastructure will do the rest: It will parse your file, pass it to your semantic highlighting feature implementation, produce a nice "golden file" pretty print of your highlighting (see the unit testing section of this document for details), and diff that result with the recorded golden file (or create it if it doesn't exist). Thus, to unit test, just create new testcases in the form of a source file, run the test once to create a golden file, look at the golden file to make sure it looks correct, and from now on you have a regression test which makes sure that your semantic highlighter always produces the same highlights for this source file.
• GSF supports embedding. Thus, if you for example implement a GSF hints provider for your language, when your language is embedded in some other files, your quickfixes continue to work; they get run, the error offsets from your AST get translated back to embedding appropriate offsets, etc.

Furthermore, for many actions, there is a LOT of code behind the scenes that is not language specific. Take mark occurrences for example. The only language specific thing there is determining, for a given offset, what the other corresponding occurrences are in terms of editor offsets. Everything else: ensuring that the parse information is up to date, listening on caret changes, painting highlights for the occurrences, defining and implementing "goto previous occurrence" and "goto next occurrence", and so on, should all be implemented in just one shared place.

One thing which is not obvious from the diagram above is that clients are free to use NetBeans APIs directly. For example, in NetBeans 6.1, the HTML module uses GSF for parsing, navigator, embedding etc. - but it registers its own CompletionProvider using the NetBeans APIs instead of GSF's code completion handler interface. It is possible to combine GSF with for example Schliemann or a custom written editor kit or custom written data loaders. More details about this are found in the custom editor kit section.

Lack of Extensibility

One of the key principles of GSF is to keep all UI in GSF. In the old days, each language support would define all its own logical editor types, color definitions, and so on. That made it really tricky to provide editor themes, as well as keep things consistent. A literal string may have one color in one file type, and another color in another filetype.

This has some implications for GSF:

• In the Java module, one of the key APIs you get is the ability to schedule a user task: run the parse tree on a Java file, and then run a task over the parse tree. (This is JavaSource.runUserActionTask()). This lets modules create arbitrary features for Java - and this is how the various Java modules (navigation, hints, etc) operate.
  
  GSF does not push this approach (it's actually there, through the SourceModel class, but you're discouraged from using it).
  
  This facility is typically used to implement specific features. And the GSF Way is to implement the feature INSIDE GSF, and expose a lean feature API to other language clients. If it's a feature needed for language A, chances are it's needed for language B as well, and putting user feature code in language plugins is the opposite approach of what GSF is trying to do.
  
  
• GSF tries to make it really easy to describe UI. It provides this through the ElementKind enumeration class. When creating code completion items, or navigation items, just reference one of the constants there, such as ElementKind.METHOD, to create an item which will look like a method, consistent across languages. This will provide an icon, a color, a code completion sorting priority etc. None of these are language specific, which is why GSF wants to provide the definitions, and language clients just reference them through the enumeration name.
  
  This of course may seem like a straightjacket - the types of code elements that GSF can address are hardcoded!!
  
  Once again, the philosophy here is that GSF is trying to address a wide swath of languages, but perhaps not every single one. Rather than saying that say a Prolog language has a particular code element that isn't provided by GSF today so Prolog should have the option of adding it, I'd like to add new types to the ElementKind enum. They can be added in every release without any backwards compatibility problems. Chances are, if something is available in one language, it's going to be needed by some other language as well. Second, there's a pretty simple workaround - pick something SIMILAR. In Ruby for example, I could have used the FIELD kind if there wasn't an ATTRIBUTE kind.
  
  Finally, there are backdoors. For example, the CompletionProposal interface (implemented by code completion items) actually has a getIcon() method, where you normally return null, but you can override the icon there. This is how I return a Ruby specific icon for Ruby keywords in code completion (and ditto for JavaScript). But overriding UI is the exception, not the norm.
  
  
• GSF lets client format the data shown in code completion items; there's the left hand side, the right hand side, as well as different colors and attributes for parameter types, return types, overridden methods, deprecated methods, etc. Rather than letting clients format HTML directly (which would for example cause color definitions to get hardcoded into the clients), GSF provides an HTML formatter to clients, and when they need to produce HTML, they use logical formatter methods to emit HTML. The actual color definitions etc. are therefore provided by GSF and clients are a bit more high level.
  
  

Service Implementations

A language plugin basically registers a bunch of callbacks. These are invoked at appropriate times by the infrastructure. There are two types of callbacks:

1. Service Implementations, such as lexing and parsing
2. Feature Implementations, such as code completion and quickfixes

• Lexing: Tokenize a document into syntactic elements
• Parsing: Parse a document into semantic structure
• Indexing: Given a parse result, store some information in a persistent (and quickly searchable) index

Lexing

GSF uses the Lexer API in NetBeans directly. All GSF languages must provide a Lexer language. It must also be registered in the Editors mime folder. See the Registration section for more details.

Parsing

GSF provides a parser interface. You can register a parser, and GSF will call your parser when needed.

NOTE: There is a new Parsing API underway. This will be something equivalent to the Lexer API, but applied to parsing. The plan is for GSF to remove its own Parser registration and interfaces and replace it with the standard Parser API. This will allow embedding and coordination not just among GSF-based languages, but for all languages that implement the parsing API (such as Java, and C/C++, which are not GSF based).

The Parser API basically just asks you to parse a given CharSequence, and return the result as your own subclass of the ParserResult interface. It is in your own parser result that you store your own AST reference. You will probably need it to implement most of the features. There is a separate document describing the parsing aspects of GSF. (Eventually you'll also want to implement incremental parsing.)

Indexing

Indexing lets you register a service which will extract information from your parse results, and store them in an index which can be queried quickly later. Your indexer will be called "at the right times" by the infrastructure:

1. At startup for all source files that are new or have changed since the last IDE session
2. After a file has been edited, and closed/left
3. Immediately if a file has been edited and somebody is trying to query the index

Feature Implementations

Semantic Highlighting

To implement semantic highlighting, you need to register an implementation of the SemanticAnalyzer interface. You will be given your own ParserResult object, and you need to return a Map<OffsetRange, Set<ColoringAttributes>>. So, you'll iterate your own AST, look for things to highlight, such as method definitions, parameters and unused variables, and for each node, look up its source offsets (start, end integer offsets in the document), and then place them in the document:

    
    // Aha, this is an unused variable reference!
    int nodeStart = node.getStartOffset();
    int nodeEnd = node.getEndOffset();
    result.put(new OffsetRange(nodeStart, nodeEnd), ColoringAttribute.UNUSED_SET);
            

Other Features

TODO: I plan to write some documents explaining how to do keystroke handling, formatting, declaration finding, code completion, etc.

GSF services, such as the lexer, parser, and feature implementations, must all be registered with the GSF infrastructure. For details on how to do this, see the Registration document.

Embedding

GSF supports language embedding - supporting nested languages, like JavaScript and CSS support inside HTML files, Ruby support (and JavaScript and CSS support) inside ERb/RHTML files, and so on. The mechanism of how this works is described fully in the embedding document.

GSF Diagnostic Tools

There is a separate module in the contrib repository, gsf.tools, which provides a number of tools in the Tools | GSF Development menu. Details are described in the GSF Development Tools document.

Unit Testing

GSF makes it really easy to unit test your feature implementations. Details on how to do this is described in the unit testing document.

Implementation Issues/Limitations

I will describe the various GSF limitations here. For now, they are listed (in very brief form) in the following Wiki page:
http://wiki.netbeans.org/GsfIssues

TODO

• Describe the philosophy of the builtin ElementKind UI, icons, fonts etc. and the general philosophy of pushing features INTO gsf rather than supporting generic user task execution.
• Describe the modules, imported and exported APIs, etc.
• Document the GSF limitations
• Write documents for specific features, like code completion and keystroke handling