Emerald Editor Discussion
March 29, 2017, 12:33:18 am *
Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
News:
 
   Home   Help Search Login Register  
Pages: 1 [2] 3
  Print  
Author Topic: I like to syntax, syntax...  (Read 32845 times)
0 Members and 1 Guest are viewing this topic.
Arantor
Site Administrator
Administrator
Master Jeweller
*****
Posts: 618



« Reply #15 on: January 09, 2007, 06:59:21 pm »

The final word on it a bit back, when it was first furiously debated was that XML would 'mostly' cover EE's obvious needs, but there wasn't really a practical way to keep it small and clean.

The obvious benefit to XML is a natural nested structure, but the structure would be only 2-3 levels deep, which makes me wonder the benefit of nesting it at all, and having it as a normal flatfile, much like CE currently uses.

I think instead of looking at the downsides of the formats, for the moment can we look at the benefits of using specific formats?

If we opt for XML, it is obviously a 'standard' of sorts and code has already been written to create and test XML files. It is also naturally more human-readable when examining the file contents.

If we opt for a custom format, we get a really tiny format (which is good if you have hundreds, or even potentially thousands, of format files). We also get a format we can design not only to meet our exact requirements in what we want to capture (be it basic highlighting, nested languages, auto complete if we wanted to), but we can tailor the code to read it really efficiently and minimise memory usage.

For the record, I am fully in favour of creating a custom format for EE. I think the benefits of a compact custom format outweigh the benefits that XML would bring, to my mind. I would admit, however, that XML is not something I am massively knowledgeable about, it is something I find out about as necessary.
Logged

"Cleverly disguised as a responsible adult!"
Matthew1344
Gem Cutter
****
Posts: 103


« Reply #16 on: January 09, 2007, 09:17:28 pm »

I also have little experience with XML, and even more limited experience with syntax files (Though I did "write" the VBS syntax file in CE).  From my limited experience, I got the impression that the flat format was very limited in its flexibility, and that XML was virtually unlimited.

For instance, a flat syntax file could tell me that "printf" is a keyword and that it should be highlighted, but can it give me a usage pattern?  If I have an object with several properties, can the flat file suggest those properties to me when I hit type "object_name" + dot (.) ?

Originally, I imagined that EE would become almost an IDE for those languages with enough detail in the syntax files--potentially providing much more intelligent help than syntax highlighting.  Since that was perhaps an over-expectation, I suppose a keyword-list-type of syntax file would be sufficient.  The IDE-type of functionality might have to be a plug-in for the particular language being used.

It's *very* possible that I'm waaaaay off on all of this... whether I am or not, I'm telling you almost *more* than I know now, so I'm content leave this in the hands of the experts.
Logged
Arantor
Site Administrator
Administrator
Master Jeweller
*****
Posts: 618



« Reply #17 on: January 09, 2007, 09:34:12 pm »

That's the beauty of having the choice of a totally new format instead of relying on a proscribed system - we can write as much or as little information as we want in our own 'flat file' format. In some respects, the final EE syntax definition might even evolve into having some of the nesting characteristics of XML.

To quote your example, there is nothing that says we couldn't set up the flat file style to cope with:
Code:
'myobject': object
methods: methoda, methodb
properties: property1, property2
classdelim: .

I'm not suggesting that the above is anything like the format we'll want to use, but it conveys in a machine readable format that an instance of myobject should be treated as an object for syntax purposes (i.e. perhaps a different colour to statements, variables etc.). It also says that such an object would have given properties, and that when highlighting, myobject immediately followed by classdelim (i.e. a period) could suggest the properties inherent to that object.

That style would also allow in the case of languages like Perl and PHP where operators such as :: and -> come into play.

Of course that could be written up in XML, but as discussed earlier in this thread, and elsewhere by Szandor, you can do all sorts of funky things like define rules.

For example, the way I read that syntax, the SDD in HTML for the Hn tags (H1 through H6) could be defined once and once only, as some form of H[1-6] rule, instead of laying out each example.

There are some examples where that would be really useful and others where it wouldn't, but having the feature there should be really good. Though for processing purposes I think it would restrict it to a strictly defined class, not the likes of . (full stop) when it came to regexp handling, as depending on length, it could be expanded on loading to be quick.

On the subject of IDE-ising things, that adds flexibility. Some programmers would like full-on autocompletion (much like Visual Studio offers), while others really hate it. The beauty is that to then add in autocompletion information wouldn't be that hard. You could almost expand it thus: (continuing from above)

Code:
'myobject': object
methods: methoda (param1 int, *param2 string, param3 bool), methodb ()
properties: property1 (int), property2 (bool)
classdelim: .

In a fraction of the space you'd need in XML, you've suddenly defined the structure of the object - the parameterisation list for each method is given (parameter name then parameter type, which can be expanded out like in Visual Studio to 'ByVal param1 as Integer' if you wanted), plus the format expectation for each property is there.

Of course for the sake of tidyness, you could either make each return type a requirement in the syntax file, or have a set of predefined ones (like int, string, bool etc) and anything else could be treated as a literal. For example when writing C using the SDL library, you will be passing SDL_surface values around, and SDL_surface is the return type. Defining it as "method_xyz (param *SDL_surface)" or whatever suggests a literal string of SDL_surface with the asterisk confirming it is being passed by reference.

Feel free to ignore totally what I'm writing here!
Logged

"Cleverly disguised as a responsible adult!"
Szandor
Senior Miner
***
Posts: 92



« Reply #18 on: January 09, 2007, 10:48:11 pm »

I also have little experience with XML, and even more limited experience with syntax files (Though I did "write" the VBS syntax file in CE).  From my limited experience, I got the impression that the flat format was very limited in its flexibility, and that XML was virtually unlimited.

SDD will not be completely flat, it is not simply an INI-file or SPC-file with a different syntax, it would be a completely new format with anything we put into it. Yes, XML is virtually unlimited but it is also harder to read and more complex to use than SDD will be. For large amounts of text XML is great, for settings it is not.

For instance, a flat syntax file could tell me that "printf" is a keyword and that it should be highlighted, but can it give me a usage pattern?  If I have an object with several properties, can the flat file suggest those properties to me when I hit type "object_name" + dot (.) ?

SDD can since it defines the entire syntax. Actually, this allows you not only to get a list of relevant keywords; it can also be used to check the document for errors, format the code automatically and even to sort order-insensitive parts of the code in any way you like. In fact, you could even set EE to check an SDD when loaded so that it's correct, using an internal SDD defining itself.

Originally, I imagined that EE would become almost an IDE for those languages with enough detail in the syntax files--potentially providing much more intelligent help than syntax highlighting.  Since that was perhaps an over-expectation, I suppose a keyword-list-type of syntax file would be sufficient.  The IDE-type of functionality might have to be a plug-in for the particular language being used.

That kind of advanced syntax definition is exactly what SDD is. Rejoice.
Logged

"Cleverly disguised as an original signature..."
Arantor
Site Administrator
Administrator
Master Jeweller
*****
Posts: 618



« Reply #19 on: January 10, 2007, 01:00:14 am »

A thought occurred to me while I was in the shower, taking the possibility of IDE-ism a bit further. Note, I haven't tried implementing this yet, but it's an idea to consider.

I figure when the SDD is loaded and parsed by EE, whatever form it takes, it must be loaded into a form of tree structure internally.

Could we not add in the ability to dynamically add to that tree structure? Of course, you wouldn't be able to amend the SDD-defined data (it'd be logged as static in the tree), but consider the following:

Code:
function somefunction() {
 // do some stuff
}

Could we not add in the ability to consider "anything between function and ( must be a function" and treat that as a user-defined keyword for this session? You'd add it to the parse tree as a function (or create a new branch for user functions if you want to highlight them differently) and when parsing the file for highlighting, you can then use somefunction as a keyword as well. Of course, it wouldn't be labelled in the same way (dynamic rather than static), because potentially you could rewrite the function name and need to cascade that change.

Additionally, if you're only loading one instance of the SDD's data per session, irrespective of the number of files that use it, you could potentially be cross-highlighting (i.e. a function defined in file1 would be highlighted in file2 provided both were open)

The only stitch-up with that is that you have to maintain said tree structure.

I realise there would be an extra processing hit, but it is a fairly unique feature if we could implement it...
Logged

"Cleverly disguised as a responsible adult!"
John Yeung
Senior Miner
***
Posts: 85


« Reply #20 on: January 10, 2007, 04:57:14 am »

Code:
function somefunction() {
 // do some stuff
}

Could we not add in the ability to consider "anything between function and ( must be a function" and treat that as a user-defined keyword for this session?

We absolutely could, if we are going the ambitious route of user-definable grammar.  (Szandor will post in about five minutes telling you how easy it is in his particular metasyntax.)  If we are doing program-constrained grammar "with lots and lots of settings" then we may or may not be able to.

John
Logged
Szandor
Senior Miner
***
Posts: 92



« Reply #21 on: January 10, 2007, 07:04:25 am »

Szandor will post in about five minutes telling you how easy it is in his particular metasyntax.

One and a half hour, actually... Also, I think my hammering is beginning to produce results since you said it before me. :-)

Anyhow, I don't know if I've gotten this straight yet. We're talking adding basic programming capabilities to the SDD? I might see why this could be cool, but I would like an example of a situation where a code specific function is handy. By defining an exact grammar we already have the potential to do quite a number of cool things with all code.

Something is forming in my brain. It's actually an undead bunny, but we can pretend it's an idé. (I'm actually thinking and writing at the same time now so excuse me if the text seem odd.)

Could an example be saving the RC-file and reloading LiteStep? Nah, this could be achieved by using macros. How about adding custom keywords? Hmm, I think that should be a feature of all SDD:s. Ok, what if I write in DocBook XML and want to gather up some statistics? The statistics would be the same for most documents, but in the case of DocBook we might want something else. How many paragraphs do we have? How many headings? How many words do we use that are only part of the main body text? It might also be nice to have a function to look for repeating words. This is not a function that will be often used so having it in the program could be unneccecary, but for large portions of text it might be nice and could instead be in the SDD for a particular syntax.

Is this the kind of things you have in mind?

I know this goes against the whole "one file to rule them all" string of thoughts, but maybe we cold have user defined external SDD functions? An SDF (Syntax Definition Function)? It would help to keep the main SDD clean from anything but metasyntax and having an SDF would be optional so it's not another case of SPC/KEY matching.
Logged

"Cleverly disguised as an original signature..."
Matthew1344
Gem Cutter
****
Posts: 103


« Reply #22 on: January 10, 2007, 03:22:07 pm »

Quote
That kind of advanced syntax definition is exactly what SDD is. Rejoice.

Hehe.  I properly described an advanced syntax definition.  Cool.   Smiley

Okay... I still don't know much about SDD, but I am convinced that you guys know what you're doing so no more XML questions.  (ooh... except... what about being able to convert syntax files using XML transformations?  Would that not be helpful?  Again, I'm in over my head here.)

I'm not sure if I'm following all of this exactly, so just to clarify...  Aranator, are you saying that EE might load the SDD, then parse the current file(s?) to obtain user-defined functions/objects/classes/etc., and use the combination as the syntax definition?  If that's what you're talking about, then....

WILL YOU MARRY ME?
 Grin
Logged
Szandor
Senior Miner
***
Posts: 92



« Reply #23 on: January 10, 2007, 05:52:42 pm »

This is how I imagine the workflow.

A user opens a file in EE. Based on the file ending, EE loads the proper SDD automatically or, in the case where the file ending is not recogniced, doesn't assign an SDD at all.

EE reads the document and compares it to the SDD. Based on the definitions in the SDD, EE can now highlight the code in the document, highlight different types of syntax in the same document (JavaScript in an HTML document for example), provide a list of keywords, warn about uncorrect code, pop-up intellisense style variables (user defined and otherwise) when proper and automatically complete and format the code. This is all possible using the basic SDD documents.

If extended with a simple scripting engine (SDF, Style Definition Functions) more advanced things are possible, like gathering syntax specific statistics, calculate parsing times and file sizes for program code, have an even more advanced intellisense and autocompletion, highlight code based on when it was or how it was edited (highlight "foo", but only if it was changed from "bar" and the file hasn't been saved) and even contain syntax specific macros that find and replace code in order to optimize it (for example, rewriting a CSS to use the shorthand format instead of several long ones). This will make EE surpass a lot (if not most) of other editors, both free and commercial.

The programmer can use EE throughout almost all of the coding process. First she will naturally write the code, using the simple text editing part. She will then proofread and format her code through the help of a basic SDD and optimize it by replacing bulky code using an SDF. The last thing she does is to use a simple macro to send the finished code to the compiler of her choice.

I must say, I didn't really like the idéa of SDF:s at first, but having written this document I think they, along with the rest of the SDD, can actually make a lot of people turn to EE for all their text writing needs.
Logged

"Cleverly disguised as an original signature..."
Arantor
Site Administrator
Administrator
Master Jeweller
*****
Posts: 618



« Reply #24 on: January 10, 2007, 10:58:57 pm »

In response to Matthew1344, that is precisely what I was getting at. From a programming point of view, the SDD must be parsed in some way on loading. This just adds a dynamic step to the process and "mini-loads" extras as it goes along, which can then be treated as if they were part of the original syntax.

In response to Szandor: the workflow you have is pretty much how it must work, at least as far as SDD goes. The idea of an SDF is appealing but would require much more maintenance to oversee than SDD on its own. However, for the majority of languages, once it has been written, it won't change all that much (e.g. C, C++), and when changes do occur, they are infrequent (e.g. PHP adding in libraries)

The beauty of the planned plugin system is that the functionality discussed would be implementable, but I don't know how applicable it would be to some languages. For mark-up related applications (like CSS) it would be very useful, but speaking as a programmer, I don't think I'd want any tool rewriting my code, simply because I've come across too many times where it hasn't worked. That said, I'm sure we can give it a try...

I really would like to see EE evolve from its CE roots and become a morphable solution, whether a minimal text editor, one with syntax highlighting and a few IDE-like features or almost a full blown IDE is what is required. That said, I know there are a lot of people very on the fence, as they like CE because it isn't IDE-like, not in spite of it. If though it can be customised and functionality removed or added at will, let's run with it.
Logged

"Cleverly disguised as a responsible adult!"
Szandor
Senior Miner
***
Posts: 92



« Reply #25 on: January 11, 2007, 09:45:22 pm »

The idea of an SDF is appealing but would require much more maintenance to oversee than SDD on its own. However, for the majority of languages, once it has been written, it won't change all that much (e.g. C, C++), and when changes do occur, they are infrequent (e.g. PHP adding in libraries)
A small note: The SDF is the simple function you suggested. I only suggested that it could (should) be kept in an external file. You're arguing against your own ideas here... :-)

For mark-up related applications (like CSS) it would be very useful, but speaking as a programmer, I don't think I'd want any tool rewriting my code, simply because I've come across too many times where it hasn't worked. That said, I'm sure we can give it a try...
The option should be there. If users like it or not is up to them.

I really would like to see EE evolve from its CE roots and become a morphable solution, whether a minimal text editor, one with syntax highlighting and a few IDE-like features or almost a full blown IDE is what is required.

I agree. That's why I like Miranda-IM, an instant messanger for ICQ, MSN, IRC, Yahoo, Jabber and just about any other protocol you can think of. The basic Miranda is ugly and can't really do much except basic chatting, but when extended with the plugins provided at their website it becomes as powerful as you care to make it.

How about this? We let EE handle the basics of SDD natively, then provide one plugin for utilizing SDD to full extent and one for those who want SDF capabilities. EE will then natively be capable of advanced highlighting, for automatic formatting and intellisense you would need a plugin and for the advanced IDE capabilities you would need another.

It would be great if the plugins could be enabled or disabled by using command line options. That way you could have one shortcut for a basic textediting version and one for the full-blown IDE.
Logged

"Cleverly disguised as an original signature..."
Arantor
Site Administrator
Administrator
Master Jeweller
*****
Posts: 618



« Reply #26 on: January 11, 2007, 11:42:16 pm »

A small note: The SDF is the simple function you suggested. I only suggested that it could (should) be kept in an external file. You're arguing against your own ideas here... :-)
Actually, I was thinking of the SDF - at least from your description - being more like tools such as HTML-Tidy, rather than highlighting extensions. I really think if we can manage it, we should include its functionality somewhere.

How about this? We let EE handle the basics of SDD natively, then provide one plugin for utilizing SDD to full extent and one for those who want SDF capabilities. EE will then natively be capable of advanced highlighting, for automatic formatting and intellisense you would need a plugin and for the advanced IDE capabilities you would need another.

It would be great if the plugins could be enabled or disabled by using command line options. That way you could have one shortcut for a basic textediting version and one for the full-blown IDE.
The SDD functionality would have to be integral whatever happened - if only for speed if nothing else. I'm not sure about leaving extended SDD/SDF functionality as separate plugins, though. Part of me thinks it would be best to integrate the functionality from day one, then plugins can take advantage of it if they want to.
Logged

"Cleverly disguised as a responsible adult!"
Soulfish
Project Manager
Administrator
Miner
*****
Posts: 37


« Reply #27 on: January 12, 2007, 10:58:16 pm »

Firstly I have to apologise for not replying to any posts in a while. Unfortunately food poisoning got the better of me and I've been recovering from that for the past few days.

With regards to the SDD and SDF - what's been discussed so far certainly looks to be some great ideas. Certainly if we implemented these ideas into EE we would have one of the best and most capable text editors on the market IMHO.

Logged

John

"A lie can run around the world before the truth has got its boots on"
Feldon
Gem Cutter
****
Posts: 106


« Reply #28 on: January 13, 2007, 02:33:10 am »

I think I missed the discussion where XML got the boot.  From the beginning, I (in my limited knowledge of such things) haven't been able to imagine why anything other than XML would be used for syntax files, but I'm sure that discussion must have taken place.  Could somebody post a link for me?

I don't know that we did officially give it the boot.  I was initially in favour of XML because a lot of people are familiar with its structure, and this might make it easier to implement in EE and to make syntax files.  However, the counter-argument is that XML was not designed specific to the purpose and therefore will likely produce larger syntax files, and will take more resources for EE to parse.  If that's the case (especially the part about taking more resources to parse) then I would say go for our own syntax format.
Logged
Szandor
Senior Miner
***
Posts: 92



« Reply #29 on: January 23, 2007, 07:13:30 am »

Inbetween my superpowered HTML coding, I try to fit some writing of the SDD. I have most of it figured out, but Istill need to work out some kinks. It's all progressing though.

By the way, we have discussed having simple programming capabilities for the SDD, what are your thoughts on this? Should we use an existing language or make up our own? I think I'm in favour of using Ruby, but I don't know what will be easiest or best to implement. Suggestions?
Logged

"Cleverly disguised as an original signature..."
Pages: 1 [2] 3
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2013, Simple Machines Valid XHTML 1.0! Valid CSS!
Page created in 0.162 seconds with 18 queries.