Trials with JPA and XPages

Intro

Don’t get me wrong, I love working with Domino and documents and all that ūüėČ , but with XPages I sometimes want to use good old fashioned SQL or even hybrid solutions. I do not want to be bothered with writing sql statements myself in the DAO classes, and I just want to use a prebuilt solution that just works. Enter JPA.

Before I go too much into this, I want to thank the guys on Slack for giving me a hand with the bits and pieces of this. I had a great deal of problems getting this thing going and it is still very far from perfect. I can at least say, however, that it runs.

Quick trial history

I started out with a simple PoC concept of trying to build a type of registry for DVDs I have. (Not creative, just quick and simple). I also decided to do this with eclipselink and built in Eclipse mars. I quickly built a few entity classes and I used the xml orm to map those entities to the database. When this was done, I exported that project to a jar file and then imported that jar into an *.nsf. This first try failed. I am sure I can think of numerous reasons why it did not work. The main issue I had was that the persistence unit I tried to configure could not be found on the runtime. At this point, I copied the eclipselink jars into the nsf directly, and copied the entities into the .nsf.  In other words, the JPA Layer was no longer its own single jar. This allowed me to try to move the configuration files around to other locations. I tried everything I could think of. Fail.

Rethink and Rework

(in other words, ask around…)

Let me just say that if you are not already a member of the XPages Slack community, why aren’t you? JOIN!!! xpages.slack.com

I went into the random channel, and posted a general question if anyone had success with JPA and XPages. Jesse Gallagher pointed out this post by Toby Samples which uses the hibernate JPA framework. I had seen this presentation before, but I must admit that it lacks the meat and potatoes needed to get it off the ground.  Dont get me wrong, it is a great resource! The other reason why I did not do much with this at first is that it was done in eclipse and not Notes Designer. Most of the stuff that I program for XPages is done directly in Designer and not in eclipse. After talking to the guys on Slack, and seeing that Toby Samples had success with hibernate, I decided to indeed give it a try.

Downloading Hibernate

In the slides (see the above link), Toby describes talks about hibernate tools being downloaded and installed in eclipse. As we all know, Notes is now based off of eclipse, albeit an older version of eclipse…¬† After a lot of searching, I did find an updateSite with the tools and I downloaded a copy of them. I then installed them onto my client as a Widget and also onto the server. This really did nothing worth while.¬† I could not access the tools in designer, nor were they available on the server. I deleted them pretty quickly. Instead, I¬†found this site to download an older version of the ORM. It is necessary to take the 4.3.11 version because Notes/Domino runs on a bitterly out-dated version of java. Once this is downloaded, I imported the required jars into my .nsf. I also put these jars into the <domino>/jvm/lib/etc/ directory as described in the slides. The only issues I had at this point was that designer couldn’t process the source quickly enough to give me the code suggestions and I had the feeling that designer was always a step away from crashing.¬† Indeed it did crash once or twice… (After a system restart it seems to have gotten better)

Configuration and Setup

The first thing that I did was to create the hibernate.cfg.xml file. This is all pretty straight forward.¬† I am also not going to discuss how this file is to be created, but I will show you my copy…

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE hibernate-configuration PUBLIC 
"-//Hibernate/Hibernate Configuration DTD 3.0//EN"
"http://www.hibernate.org/dtd/hibernate-configuration-3.0.dtd">

<hibernate-configuration>
   <session-factory name="hibernateSessionFactory">
        <property name="hibernate.connection.driver_class">com.mysql.jdbc.Driver</property>
            <property name="hibernate.connection.password">****</property>
            <property name="hibernate.connection.url">jdbc:mysql://192.168.0.1:3306/library?createDatabaseIfNotExist=false</property>
            <property name="hibernate.connection.username">DB_Programmatic_User</property>
            <property name="hibernate.show_sql">true</property>
            <property name="hibernate.dialect">org.hibernate.dialect.MySQLDialect</property>
            <property name="hibernate.search.autoregister_listeners">false</property>
            <!-- <property name="hibernate.hbm2ddl.auto">create</property>  -->

            <mapping class="de.domain..bluemix.jpa.entities.Actor"></mapping>
            <mapping class="de.domain.bluemix.jpa.entities.DVD"></mapping>
            <mapping class="de.domain.jpa.entities.Genre"></mapping>	
      </session-factory>
      
      
</hibernate-configuration>

The second thing I did was create an application listener with static information for creating sessions.

package de.domain.mysqltrial.services;

import java.io.Serializable;

import org.hibernate.SessionFactory;
import org.hibernate.boot.registry.StandardServiceRegistryBuilder;
import org.hibernate.cfg.Configuration;
import org.hibernate.service.ServiceRegistry;

import com.ibm.xsp.application.ApplicationEx;
import com.ibm.xsp.application.events.ApplicationListener2;

public class PersistenceManager implements Serializable, ApplicationListener2 {

      private static final long serialVersionUID = 1L;

      private static SessionFactory sessionFactory;

      private static boolean init = false;
      
      public static SessionFactory getSessionFactory() {
            if((sessionFactory == null) || (sessionFactory.isClosed())){
                  throw new IllegalStateException("Session Factory is null or closed!");
            }
            return sessionFactory;
      }
      
      public static boolean isInit() {
            return init;
      }
      
      private void init(){
            if(!isInit()){
                  try{
                        System.out.println("Initializing Session Factory");
                        
                        Configuration conf = new Configuration().configure();
                        ServiceRegistry serviceRegistry= new StandardServiceRegistryBuilder().applySettings(conf.getProperties()).build();
                        sessionFactory = conf.buildSessionFactory(serviceRegistry);
                        init = true;
                  } catch(Throwable t){
                        t.printStackTrace();
                  }
            }
      }

      public void reInit(){
            destroy();
            init();
      }
      
      private void destroy(){
            System.out.println("Destroying Entity Manager Factory");
            init = false;
            if(sessionFactory != null)	sessionFactory.close();
            sessionFactory = null;
      }

      public void applicationCreated(ApplicationEx arg0) {
            init();
      }

      public void applicationDestroyed(ApplicationEx arg0) {
            destroy();
      }

      public void applicationRefreshed(ApplicationEx arg0) {
            reInit();
      }
}

The point of the application listener is to make sure clean up is done correctly and to make sure that everything is initialized correctly.  It probably is not needed in this way, but I found it at the very least to be a cool idea. This class must also be registered. Here is a screen shot with the location of these files.

package

 

 

 

 

 

Primary Problem

This configuration worked…. almost. I kept getting security exceptions. The runtime was not being granted the permissions it needed to run the code. Only after I added the following lines to the java.policy document was I able to get the code to execute properly.

permission java.lang.RuntimePermission "getClassLoader"; 
permission java.lang.RuntimePermission "setContextClassLoader"; 

permission java.util.PropertyPermission "jboss.i18n.generate-proxies" "write"
permission java.security.AllPermission;

This is a situation that I find sucky. It is alright for a test environment, but I would not want to mess with the policy file for an end-customer. My question is, does anyone have a possible solution for this?

Conclusion

It is possible to use JPA and it works well,  but I am not happy about the wide-open security window that was necessary in order to get it to work.

One more time, thank you to those on Slack who gave me a hand…

Toby Samples, David Leedy, Jesse Gallagher, and others who had hints…

 

Attachments and Java Beans

Up until this point, I must admit that I have been lazy. Even though most of the XPages I have created in the last two years have made extensive use of Java Beans, I have left the attachments to the XspDocument and the typical upload and download controls. I did not want to open that can of worms and just wanted to stick with what I know works. Well, that is dumb. It is fine for one or two minor applications that are never going to be used anyway, but when it comes down to it, I want it to be correct. Today, I started that adventure and, with all new things, a google search was performed for anything that could help me and point me in the correct direction. (Honestly, what did you old folks do without search engines? I’d be lost! Or at the very least spending 10 hours a day at the public library!) What I did not find was a document that contained upload and download information. Since I do not want to loose what I found today, I decided to write a quick post. Thank you to everyone that I am stealing from to write this…. ūüėõ

The first thing that I noticed was that my concept was faulty. At least I think it was. I wanted to have a single file in my Java Bean that I could upload and download at will and access and save in my DAO layer. Of course I could be mistaken, but it does not seem to work that way. Uploading documents and downloading them again needs to be performed in two different actions, and in two different ways, and with different objects. Futhermore, I do not even offer both functions on the same page, though both are possible with the same bean.

First off, my test bean is very simple. If I were to extract an interface (just to get a quick look at what the bean contains, it would hold the following information

public interface IFileTest {

/*
* This function contains the information to save the document. Normally, I do this in a seperate DAO layer Object.
* The example that I used had the majority of the information in one single class.
* I did not experiment as I wanted to keep everything as simple as possible. Such experiments are further on my to-do list.
*/
public abstract void save() throws Exception;

/*
* This function contains the logic to download the attachement. It is performed in a separate XPage containing only this function call.
*/
public abstract void downloadAttachment() throws Exception;

/*
* This function returns a string that points to the xpages with the download attachment function call.
*/
public abstract String getDownloadURL();

/*
* This function will read the parameters from the URL in order to initialize the data for the viewscoped bean.
* Normally with the Beans I create, this function will access the DAO and set the data in an object contained by this Bean.
* This Bean is a controller in the MVC design pattern.
*/
public abstract void init();

/*
* I just find this helpful.
*/
public abstract String getUnid();

/*
* com.ibm.xsp.component.UIFileuploadEx.UploadedFile. This object is used ONLY in the upload core XPage control.
*/
public abstract UploadedFile getFile();

public abstract void setFile(UploadedFile file);

}

 

As i said, this test is done with as simple a construct as possible.

After this was completed, I worked on uploading a document. It seemed the most logical starting point. My primary source for this was a StackOverflow question posted by David Leedy so, note that the following code is primarily coming from Mark Leusink, the accepted answerer of Mr. Leedy’s question. The first part is the most simple. I have a property in my Bean that is of type com.ibm.xsp.component.UIFileuploadEx.UploadedFile . I have a corresponding getter/setter pair. I use EL to link the core control for uploading data to the bean. The real magic happens in the save logic.

public void save() throws Exception{
    
    /* I use the openNTF Domino API (ODA) for nearly all of my
     * applications. If this was not the case, we would have to worry about
     * proper recycling. Keep in mind that this is also just a test. Normally
     * my routines have much cleaner error handling.
     * 
     * The following statement uses a utility that I built that helps me get
     * key objects. I am assuming here that you know how to get a handle on the current
     * document. 
     */
    Document doc = ODASessionHelper.getCurrentDatabase().createDocument();
    doc.replaceItemValue(FIELD_FORM, FORM_NAME);
    
    // file is and instance of UploadedFIle. This is the Property that is bound to the core FileUpload control
    if(file != null){ 
      IUploadedFile fl = file.getUploadedFile();
      
      //File is the standard java.io variant.
      File file = fl.getServerFile();
      
      String fileName = fl.getClientFileName();
      // this gave me ONLY the name of the file without any path information.
      System.out.println(String.format("clientFileName: '%s'", fileName)); 
      
      // on my system, this gave the character ";"
      System.out.println(String.format("seperator is '%s'", File.pathSeparator));
      
      // This gives you the location of the file that was uploaded by the control on the server.
      File realNameFile = new File(file.getAbsoluteFile() + File.pathSeparator + fileName);
      System.out.println(String.format("realFile name: '%s'", realNameFile.getAbsoluteFile()));
      
      boolean renamedFile = file.renameTo(realNameFile);
      if(renamedFile){
        //typical code to attach a file to a document.
        RichTextItem body = doc.createRichTextItem(FIELD_BODY);
        body.embedObject(EmbeddedObject.EMBED_ATTACHMENT, "", realNameFile.getAbsolutePath(), null);
      } else {
        throw new Exception("file could not be renamed");
      }
      doc.save();
      
      /*
       * Normally at this stage, I save the UNID so that I get that document again
       * to prevent a bunch of new documents being created.  This is just me being 
       * lazy and wanting to get a test out ASAP.
       */
    } else {
      throw new NullPointerException("file was null");
    }
  }

The only issue that I have with the above code is that the new name of the attachment is a bit messed up. I do not know if it is because the operating system is windows, or if it is because of the domino version, but the attachment name is changed to “_<strangenumbers>tmp;realAttachmentName.txt” . This is because File.pathSeperator is a semicolon. I have a workaround for this in my download function, but a workaround is still only a workaround.

As I previously said, I did not find a post with both upload and download functionality explained. I did find an awesome article on openNTF regarding downloading attachments programmatically. So, here is a quick shout-out to Naveen Maurya who posted the XSnippet. In the example provided, an XPage was built which called a server-side JavaScript function which got a handle on the FacesContext and the server response to download all files in a zip file. I just edited this to be run in my Bean and not in JavaScript.

/*
   * same disclaimer. I wanted to this quickly. Normally my error handling is 
   * significantly better. I just want the theory here.
   */
  public void downloadAttachment() throws Exception{
    Database db = null;
    Document doc = null;
    
    //java.io.OutputStream;
    OutputStream stream = null;
    
    // java.util.zip.ZipOutputStream;
    ZipOutputStream out = null;
    
    // java.io.BufferedInputStream;
    BufferedInputStream in = null;
    
    try{
      if(StringHelper.isNullOrEmpty(getUnid())) throw new IllegalStateException("Unid is null");
      
      // again, I am using ODA, and this is just a way to get the current database.
      db = ODASessionHelper.getCurrentDatabase();
      
      /*
       * I normally do this in multiple steps.
       * 1. try to get the document with the UNID
       * 2. try to get the document with the noteID
       */
      doc = db.getDocumentByUNID(getUnid());
      if(!doc.hasItem(FIELD_BODY)){
        throw new IllegalStateException("body not located");
      } else {
        Item item = doc.getFirstItem(FIELD_BODY);
        if(!(item instanceof RichTextItem)){
          // I would assume that I would have to come up with a MIME variant as well.
          throw new IllegalStateException("item is not of type richtext");
        } else {
          // normally I ask if item is instanceof RichTextItem
          RichTextItem body = (RichTextItem)item;
          
          Vector objs = body.getEmbeddedObjects();
          if(objs.isEmpty()){
            throw new IllegalStateException("body has no objects to download");
          } else {
            
            ExternalContext extContext = FacesContext.getCurrentInstance().getExternalContext();
            // javax.servlet.http.HttpServletResponse;
            HttpServletResponse response = (HttpServletResponse)extContext.getResponse();
            
            response.setHeader("Cache-Control", "no-cache");
            response.setDateHeader("Expires", -1);
            response.setContentType("application/zip"); // change this for different types.
            // I gave a static name to my zip file, but the original code was dynamic
            response.setHeader("Content-Disposition", "attachment; filename=Attachments.zip");
            
            stream = response.getOutputStream();
            out = new ZipOutputStream(stream);
            
            for(EmbeddedObject att : objs){
              in = new BufferedInputStream(att.getInputStream());
              int length = in.available();
              byte[] data = new byte[length];
              in.read(data, 0, length);
              String nm = att.getName();
              
              /*
               * This is my workaround for the file names. Although they are saved in the document
               * with the incorrect name, I could at least download them again with the proper name.
               */
              ZipEntry entry = new ZipEntry(nm.contains(";") ? StringHelper.rightSubstring(nm, ";") : nm);
              out.putNextEntry(entry);
              out.write(data);
              in.close();
            }
          }
          // cleanup should be done properly.  this is a 'do as I say, not as I do' moment.....
          out.flush();
          out.close();
          stream.flush();
          stream.flush();
          FacesContext.getCurrentInstance().responseComplete();
        }
      }
    } catch(Exception e){
      // very nasty error handling....
      e.printStackTrace();
      throw e;
    }
    
  }

In conclusion, I have a test XPage application with one form, and with two xpages. The one xpage allows saving attachments. It has the File Upload control available by the XPages core and a save button. The second XPage is only used for downloading the attachments. It holds no content, but gets the file to download via the HTTPServletResponse in the beforeRenderResponse XPage action. The UNID of the document is passed with the URL.

Although not implemented in an xpage, I also built the logic to open the URL in a new window using client side javascript:

window.open("#{javascript:FileTest.getDownloadURL()}", "_blank");

FileTest in the above example is the name of the bean as configured in the FacesConfig.xml file.

My next steps would be

  • to build a view with which I could display the file names and other typical information available for file downloads
  • export files without being compressed into a zip file
  • it goes without saying that I would have to refine the above functions and build in proper cleanup and error handling

Happy Programming!!




Memory: A little fucker that dies before its time AKA Crap-Filled bowling balls

As anyone who knows me can tell, I take great pride in every application that I work with.¬† Since spearheading my company’s XPage development starting in 2010(ish), I have developed, analyzed, and fixed numerous apps.¬† They are like little children that send off into the real world.

So when I get reports that one of them is misbehaving, I get real defensive, real fast. It is, unfortunately, my downfall. However, every app that I need to re-evaluate is a learning potential and I do treat it as such.  *Spanks the bad app with a vegance*  (Ok, not really)

Bad jokes and terrible ideas later, I will get into the issue at hand.

Lets call this app ‘Waterfall Workflow’ or WWF. WWF is an application where I can expect a peak of about 800 users concurrently at its absolute extreme maximum. Normal operations should be about half of that number. Users sign in to a main application which holds no more than configuration information and which is responsible for the XPages and coding. Coding is done primarily in Java. All code uses ODA, or as it is officially named, the openNTF Domino API, or just THE API!!! (It all depends on who is speaking ūüėõ ) Hi Paul, David, and Nathan!!! It also makes heavy use of the Extension Libraries, but lets forget that for a moment.

The data is contained on about 4 separate .nsf databases. Each database has a specific function, i.e. Labels and Languages, Primary Data, Global configurations and database instances, etc.¬† Because I do not want to build a database connection for every little piece of the puzzle, I lazily add every piece of configuration heaven into a cache in the application scope.¬† This is done through a series of ‘Controller’ type java classes.¬† No worries, I do not load every possible piece of scrap into its own AS variable.¬† Everything is neatly organized!!!¬† (Your pride is showing….¬† oops *zip*) The primary data is obviously not cached…. why should it be….

So all is fine and dandy until I decide to build an advanced search. Should be easy, right???  Yeah, why not. So lets look at my solution and take a look at some more specifics of WWF.

  1. We are dealing with an application with heavy Author/Reader field usage.¬† (well there goes performance, but there is not too much I can do there…. I think…)
  2. We are dealing with approximately 60,000 documents worth of primary data. (Remember other information is stored in cache as fetched from other nsfs)
  3. Each primary data document may hold a single attachment, and every primary data document may have a separate document (linked via a key) containing up to 10 attachments. This gives us a max total of about 120,000 possible documents where the actual value is likely closer to roughly 80,000.
  4. The search is done in such a way that the query could have 20,000 hits or more. (theoretical)

Productive Server Info

  • 2 XPage applications are run on this server.
  • we are dealing with 64 bit Windows and 64 bit Domino 9.0.1 FP3 and some odd-numbered hot-fix pack
  • October 2015 ExtLibs, and ODA

The implementation of the advanced search is a pretty simple. A user gets the possibility to select values for up to 10 or so fields contained in the primary data document.¬† (There are a total of about 120 odd fields per document) Dependent upon the users’ selection, a DbSearch is performed after building a query string.¬† Although I cannot remember why, I know that a full text index of the database was not built, and one is not desired.¬† The DbSearch is set to return a maximum of 500 documents. Depending on the selection of the user, a further search is performed on an archive which contains old data.

As previously stated, all actions performed using domino data is performed using THE API (ODA). This of course includes the search. Once the search delivers the document collection, an iterator is created over which the documents are read out one by one into java objects which is then stored in a list.¬† These java objects contain roughly 15 string attributes, and we are talking about a maximum of 1000 returned documents. (2 searches, each returning 500 documents)¬† This is nothing ground breaking. This list is stored in a session scoped controller (so that the view can be closed and re-opened without performing a search a second time). We found no issues testing with up to 10 people in the testing environment.¬† We let this functionality go live, and BAM!!!!!!!!¬† OutOfMemoryErrors hit us (ok, hit me) like a ton of crap-filled bowling balls and I still cannot get the stench off of me. Design restore. Wash. Rinse. Rethink…..

Since the design update included numerous little changes, I first had to localize the problem.  JMeter to the rescue in our confined QA environment which (as far as I can tell) is a 1 to 1 mock up of the final server. Same OS, same hardware specs, same (at least where it counts) config.  Same OSGi Plug-ins.

After setting up a test plan where x dummy users login, go to the search page, submit a search request via AJAX, I thought it would be a good idea to set x to 100 users. (All of which are using the same credentials, but by checking the cookies they are all on their own individual sessions) No more than 10 search requests were submitted before BAM!!!!!  Another ton of crap-filled bowling balls.  Server restart, Wash, Rinse, Repeat.

So, where am I going wrong then?

I quickly build another app in the QA system containing one xpage, no configuration cache, and only a DbSearch and a dummy java object being saved in the session scope. So far, only ODA was tested, and the same function construction was emulated. (obviously without the extra final version finesse) Same problem.  Next step, find out which step in the code is causing the error, and by the way, lets cut it to a simple 5 or 10 dummy users.

Before I go further, I want to explain the princess that is the JVM. She has maximum memory (this is how big her brain is), she has an available memory (how much she is willing to give you at the moment, but she’ll give you more if you need it), and a used memory (how much she is actually thinking about your lovely self). Lets expand this into the domino world, and we have two notes.ini variables with which we can play. HTTPJVMMaxHeapSize and its buddy HTTPJVMHeapSizeSet (or whatever). On a 64 bit system, you can play with this a bit. Its default is 256M, referring to a total maximum runtime memory of 256MB, and its buddy, when set to 1 (as far as I know) tells domino not to reset the max heap size.¬† Don’t quote me on that though, it has been a while since reading Paul Wither’s awesome XPage book.

After every critical call, and after every 100th document being iterated over, I printed:

  1. free memory calculated to MB
  2. total available memory calculated to MB
  3. maximum memory

From beginning on, I only had about 35 MB of memory, 64MB available, and a total of 256MB. I played with the setting, going up to 512MB, and then a total of 1024MB. I found a few interesting things:

  1. Viewing the task manager resource/performance panel, the memory usage on the server never exceeded roughly 4 GB of the available 16 GB RAM.
  2. The available memory never exceeded 64MB
  3. the free memory (ok, i obviously was not seeing every milli-second’s value), never went below 5MB.
  4. On the server console, the iteration looping continued although I was also reading the bloody OutOfMemoryError crap-filled bowling ball message.

I am left with an interesting challenge.¬† What is the cause of this stupid bowling-ball shower? The following thoughts are going through my head…

  1. Is a domino configuration setting messing with me, and is that why the available memory is not increasing to match my current needs?
  2. Am I doing something wrong with the loop?
  3. Is it possible that the problem is not with me, but with my tools?
  4. Is it possible that ODA cannot recycle the objects fast enough to handle 10 concurrent requests to perform a function which does a dbsearch over approximately 80,000 documents and returns a maximum of 500?
  5. Is it possible that the OSGi Runtimes are not getting the memory they need to run?  If not, why would that not take the same value as is written in the notes.ini?
  6. What the fuck am I missing?
  7. How do I get the smell of crap-filled bowling ball of me?  Does tomato juice work?

As you can tell, I am still trying to figure this out. I don’t expect you to have learned anything from this, but at least I got my thoughts out.

I am going to try taking yet another shower.

 

Enter the OSGi – Cool shit is on the way

holistic-net has a bunch of little projects/products, all of which has something to do with IBM Connections – a sort of business social website for either intra- or internet. ¬†Greenhouse is a good example of it. ¬†In order to bunch our products and services which do something with Connections, we are looking into transferring those products/projects into a completely modular java project. ¬†The framework¬†of choice — Eclipse 4.

I am new to eclipse and OSGi¬†development, even though I started to dabble in extending the XPage runtime with OSGi, I am not ashamed to admit that I was not sure what I was doing… Those of you on stack overflow more than likely know that I have asked many questions regarding OSGi dev and I have not been overly successful. ¬†I am ashamed to admit that I did not first look into how eclipse development really works and decided instead to jump off the deep end and somehow expect it all to work. ¬†Lesson learned.

Come to think of it, I learned a great deal in the past two weeks.  So much so that I am about to burst.  Moreover, today is one of those days that shows why I want to be a programmer, why I love my job, and why I cannot imagine doing anything else.  In the past two weeks I was frustrated with example projects, fearful that I will not be able to learn the concepts and get things going, and perhaps a bit down on myself.  Today was the pivotal moment when it all went click and I can say that I am not only happy with what I have accomplished, but I am also seeing the full power of the eclipse framework.  Even more than that, I think I can take what I learned with plug in management and organization and move it into all of my future project development.  (Although that is more than likely just personal preference stuff anyway)

Ok, so none of this is making much sense to “normal” people so I guess I better explain myself.

1. Separation of UI development and data management.
What I mean by this is that all you need is a data model.  This model is best in an Interface.  Aside from this, you can have a service model interface.  These two interfaces are then used to build the UI.  Actually, what I have been doing is using the model interface to build a jface bean.  There seems to be a bit more to it than building a Bean for XPages, but not too much more.  In fact, it is pretty easy when it comes down to it.  Just build an abstract bean model and then just extend it and add a bit more code into your setX methods.  This allows the GUI to not care at all where and how it is getting its data.  It only wants it to follow the interface that is located in the model plug-in.  This allows for easier GUI testing, and the gui does not have to be updated if the Data Services change.

2. Modular Design
Imagine you have two brothers. ¬†Both of them are going to use the same budgeting software programs that you design. ¬†Brother one, lets call David, is not too bad with computers, has no fear of it, and can follow your install directions, even if they are a bit more complex. ¬†For him, you want to use your default design of saving information into a database. ¬†This is the easiest way to keep track of and query your data, so why not. ¬†You can deliver the project easily to him with the data service plug-ins necessary for JPA data persistence. ¬†And since your data is saved in a model, you can use the service Interface to verify that you are delivering the proper functions. ¬†Now let’s say you have a second brother named Brian. ¬†We can admit that he is a bit of a douche and does not like fancy stuff. ¬†He likes his toys to be slow and if he can make you run around the block a few times, he will gladly go inside, get a lawn chair and a lemonade, and he will watch you jump through hoops on that blistery summer afternoon. ¬†You can build another service that follows the model service interface and it will handle the data the way Brian needs. ¬†Later, when John comes along, he can decide if he wants to be a Brian or a David, you still have both service implementations and he can decide which he wants to use.

We can even take this a step further and not limit it to our annoying brothers.  We can also look at pure content.  We have an application and we want to define different roles to it.  The first role is purely user input and basic displays.  He can have an instance installed onto his computer containing the plug-ins that allow him to most efficiently perform his task.  The second role is more of an administrator.  He needs to be able to do everything that first role can do, plus this user needs to be able to perform administrative tasks and needs advanced perspectives to handle those tasks.  His instance of the application will include those plug-ins all of which extend the base UI.

3. Styling controlled through CSS
Actually, I am not sure whether or not this was an option in normal SWING UI development.  I do not think so.  Anyway it makes customizing the front end very easy.  I do not know too much about this yet, but I see it as a cool ability.

4. Event Driven application with contexts and a lot of cool shit like injection
The framework takes care of all of your application needs.  Set an object into the context and be done with it.  If the object is configured to be available, the framework will also inject it into your methods and you do not have to worry much.  (You need to be smart, but still)  This eliminates the need for a single factory class or program class that maintains critical program data. There is so much candy that I would have to write a book to get into it all.  I suggest you try it yourself.

 

This means that cool stuff is going to be done with BudgIt, or well MoneySucks as I have now started to call it.  I will be transferring everything into an Eclipse 4 application.  I am excited and cannot wait to get started.  Come to think of it, I might as well start downloading eclipse right now!

Beginning with Java

Welcome to the Java Tutorial

Welcome to a new tutorial of learning how to program with java!  I have tried to make videos of myself doing all of this, but I think this will work out better.  It is my goal in this tutorial to teach anyone interested the basics of programming.  We could start in any language, but I feel that Java offers the best structure to learn Object Oriented Programming.

Object Oriented Programming (OOP) is a way to organize a program into objects.  Java makes OOP a founding philosophy that creates a language that is very robust and expandable while making a complex program easy(er) to understand.  Granted it could be said that java just makes simple tasks very complex and that would not be wrong.  But it is also a very large leap forward from C++.

Java is a very interesting language.  A few advantages are:

  • forced Exception (error) Handling. ¬†Any error that is created and known about must be handled at design time or else the program will not “compile”
  • Strict OOP programming style
  • platform and operating system independance
  • well documented
  • plenty of online tutorials
  • free Integrated Development Environments (IDEs)
  • easy switch to other OOP languages such as Microsoft’s C#

Numerous programs have been created with Java, including games. Minecraft is one example, but there are plenty more such as Applets that run on browsers that aide in seat selection at an online ticketing service, programs that automate coffee machines, parking machines, and plenty of others.  There is no better language to start with, not even, in my opinion, Visual Basic.

Let’s set up our environment!

In order to start our programming tutorial, a few programs must be in place.  The first step is to get the Java Runtime Environment (JRE).  In many cases, this is already installed.  The easiest way to verify this is to open a command line window and give the java -version command

java -version command

java -version command

If the screen looks like the window above, regardless of which java version is displayed, than a JRE is already installed and you need not worry about downloading it.  Otherwise, it must be installed.  To do this, go to the following webpage and search for the appropriate JRE install file:
http://www.java.com/en/download/index.jsp
It is also easy to find by putting JRE Download into google.

The next step would be to download the java development tookit (JDK).  That can be done at the following website
http://www.oracle.com/technetwork/java/javase/downloads/index.html
Again, only a simple google search is needed.  All you need is the standard edition.  It is not necessary to sign up or register for any of these downloads.

The third download is also very easy to find, and it will not be “installed” into the system, just started. ¬†For this tutorial series, we will be using eclipse Juno. ¬†Currently the newest release is Eclipse Kepler, but the two are almost identical and it is not important to have the same version as I have. ¬†You can download eclipse from here:
http://www.eclipse.org/downloads/
I often download the EE version, but the standard is just as well for getting into java for the first time and for mostly all projects.  Once this file downloads, extract it into a place that it can stay for a longer period of time.  I do not recommend the desktop, but that is your decision.

Once that is finished, we will want to start eclipse.  To do that, just open that folder and open the eclipse executable that has a purple icon:
EclipseFolderA window will open up where you can select your desired workspace.  The default will be located in your personal documents folder.  You can change this if desired, or even create a new folder easily.  I started out creating a new workspace for every program I worked on, but this is unnecessary.
eclipseWorkspaceWith these steps done, eclipse will start and it is possible to start programming.  In the next tutorial, we will look at where to go from here, including looking at a few basic types in Java, and creating a program that allows us to put information into the user console.

 

Java and Encryption

Welcome to a small tutorial like lesson where I go into a bit of encryption etc in Java.  This will not be a highly detailed discuss, I will just go into the basics.

First off, what is encryption?
Cryptography is, in its most basic form, preparing information for transmission in such a way that it cannot be read or understood by others.¬† Cryptography got its start thousands of years ago, and has taken on may forms.¬† My favorite historical example is a method whereby a piece of “paper” could be wrapped around a stick. ¬†The message would then be written on the paper. ¬†Once unwrapped, it would be difficult to know what it was saying without knowing the diameter of the stick that was used to “encrypt” the message, if you could tell what it was in the first place. ¬†It is said that the Spartans used this system. ¬†Later, another encryption system was used where the letters would be transposed a few spaces. ¬†In other words, ‘A’ would become ‘C’, ‘B’ to ‘D’ and so on. ¬†Current methods are not too different.

Why am I using encryption in BudgIt?

My use case is fairly simple.  I want to be able to store some basic program information that the normal user should not be able to see or change without my business logic.  I need the information available without a database connection, so that would not do, and although I could use some sort of a binary reader/writer, the point of BudgIt is to learn, so that is what I am going to do.

Encryption would probably be better if the goal is to transfer data across the internet. ¬†This would hopefully prevent eavesdroppers from getting personal and sensitive information. although you still have to worry about the NSA. ¬†ūüėÄ ¬†In fact, it has even been reported that they have overseen the development of a few ciphers in order to make sure that they have a “master key” that can unlock any encrypted transmission. ¬†But alas: politics is not the reason for this post, so lets just jump into it.

Lets create a key in java

private void createKey(){
		String msg;
		KeyGenerator keygenerator;
		SecretKey key;
		String keyfile = FILE_KEY;
		ObjectOutputStream oos = null;
		File f = new File(keyfile);
		if(f.exists()){
			//Warning: Key already exists!  Would you like to generate a new key?  This could make information that already exists unreadable.;

		}
		try{
			keygenerator = KeyGenerator.getInstance("DES"); // this says to use the DES algorithm when it creates the key file.
			keygenerator.init(new SecureRandom());
			key = keygenerator.generateKey();
//initiate the key generator and create a key
			SecretKeyFactory skf = SecretKeyFactory.getInstance("DES");
			Class<?> spec = Class.forName("javax.crypto.spec.DESKeySpec");
			DESKeySpec ks = (DESKeySpec) skf.getKeySpec(key,  spec);

			oos = new ObjectOutputStream(new FileOutputStream(keyfile));
			oos.writeObject(ks.getKey());

			Cipher c = Cipher.getInstance("DES/CFB8/NoPadding");
			c.init(Cipher.ENCRYPT_MODE, key);
			oos.writeObject(c.getIV());
			oos.close();

		} catch(NoSuchAlgorithmException e) + many others that I will skip{
			//The encryption algorithm DES is unknown to this system.
		} finally{
			if (oos != null){
				try {
					oos.close();
				} catch (IOException e) {
					e.printStackTrace();
				}
			}
		}
	}

This is a very complex bit of code to wade through and I do not really want to rip it to pieces explaining what each part does, but I will say this.  This bit of code is creating a key and saving that key to a file to be used later by an encryption and decryption function.  It is important to note that if this key is lost, the files saved using this key will no longer be able to be read.  (unless you are the NSA).

 

Next, we are going to use that file to encrypt an array of strings.  Again, I am not going to go line by line through it, but it is pretty self explanatory.  It looks more scary than it really is.

public void encryptToFile(String fileName, String[] strings) {

		// check existance of the key
		File f = new File(FILE_KEY);
		if (!f.exists()){
			//An encryption key could not be found!  
		}

		ObjectInputStream ois = null;
		PrintWriter pw = null;

		try{
			ois = new ObjectInputStream(new FileInputStream(f));
			DESKeySpec ks = new DESKeySpec((byte[])ois.readObject());
			SecretKeyFactory skf = SecretKeyFactory.getInstance("DES");
			SecretKey key = skf.generateSecret(ks);

			Cipher c = Cipher.getInstance("DES/CFB8/NoPadding");
			c.init(Cipher.ENCRYPT_MODE, key, new IvParameterSpec((byte[])ois.readObject()));
			CipherOutputStream cos = new CipherOutputStream(new FileOutputStream(fileName), c);
			pw = new PrintWriter(new OutputStreamWriter(cos));
			for (String s : strings){
				pw.println(s);
			}
			pw.flush();
		} catch long list of exceptions {

		} finally{
			if(ois != null){
				try {
					ois.close();
				} catch (IOException e) {
					e.printStackTrace();
				}
			}
			if (pw != null){
				pw.close();
			}
		}
	}

Here we are decrypting a file that we already saved.  This must use the same key that we used to encrypt the file.

 
public ArrayList<String> dycryptFile(String fileName) throws FileNotFoundException, IOException{

		ArrayList strings = new ArrayList();

		String msg;
		// check existance of the key
		File f = new File(FILE_KEY);
		if (!f.exists()){
			//An encryption key could not be found!
		}

		ObjectInputStream ois = null;
		CipherInputStream cis = null;
		BufferedReader br = null;
		String input = "";

		try{
			//********************************************** get cipher**********************************
			ois = new ObjectInputStream(new FileInputStream(f));
			DESKeySpec ks = new DESKeySpec((byte[])ois.readObject());
			SecretKeyFactory skf = SecretKeyFactory.getInstance("DES");
			SecretKey key = skf.generateSecret(ks);

			Cipher c = Cipher.getInstance("DES/CFB8/NoPadding");
			c.init(Cipher.DECRYPT_MODE, key, new IvParameterSpec((byte[])ois.readObject()));
			// ************************************************* decrypt file********************************

			cis = new CipherInputStream(new FileInputStream(fileName), c);
			br = new BufferedReader( new InputStreamReader(cis));

			while ((input = br.readLine()) != null){
				strings.add(input);
			}

		} 
		//... catch long list of exceptions ...

		} finally {
			if(ois != null){
				try {
					ois.close();
				} catch (IOException e) {
					e.printStackTrace();
				}
			}
			if(br != null){
				br.close();
			}
		}
		return strings;
	}

This is just a quick block to get all of you interested people an example to look at.  I do hope to write again soon and go into each of the methods and say what each part is actually doing and why it is important, but for now I hope that it can be understood.