Developer Forum

Showing: limited to 100 messages
MySQL Decrementing Dates
bront 2018-10-12 13:00

hi,

I am running into an issue using MySQL as an external data source. Dates are being decremented.

For example, the correct date 2018/10/12 in the database appears as 2018/10/11 19:00:00 in the JSON response.

Here is a example of the connection string we're using:

jdbc:mysql://database:port/schema?autoReconnect=true&useUnicode=true&characterEncoding=utf8&zeroDateTimeBehavior=CONVERT_TO_NULL&useSSL=true&verifyServerCertificate=false&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC

I am able to connect from my local machine using this string and successfully pull the correct date.

Any ideas on how to fix this?

Many thanks,

bront

view message
Insert User Data
WayneH 2018-10-09 06:40
Is it possible to bulk insert user data to the core user schema?

thanks

Wayne H
view message
Announcement: DataIntegration module move from svn to git
tgaluhn 2018-09-25 16:26

As previously announced, in the upcoming 18.3 release of LabKey Server, the DataIntegration (ETL) module will be a premium feature to support LabKey continued investment in its development. To support this, effective with SVN trunk commit r60372, the module has been moved from LabKey’s SVN server into a private, LabKey-managed GitHub repository.

This will impact you if:

  • You have a local development box
  • You are pulling source code from the trunk in SVN, or will be pulling source code for 18.3 or subsequent releases
  • And at least one of the following is true on your development machine:
    • You are writing ETL xml definitions in a module
    • You are running ETLs defined by any module
    • You are writing user defined ETLs in the application

If you will be affected by this change and have not already been contacted, or if you have any other questions or concerns, please contact your LabKey representative.

Thank you!

view message
A helper for using the python API
Brian Connolly (Proteinms.net) 2018-09-20 16:10

One of the big annoyances when using it the LabKey python api is that I need to define the URL for the LabKey server as a bunch of different variables. This makes it hard to write scripts/tools for non programmers to use.

This problem has gotten so annoying lately that I wrote a little function to help me The function simply takes a LabKey URL and breaks it into it's component parts. It has been so useful I figured I would share it.

The function which will take a URL and

  1. Check if it for a LabKey Server
  2. Rip the URL apart and find the hostname, port, protocol, context path and container path.
def read_url(url):
    """
    Break apart URL provided into protocol, hostname and context path
    :param url: LabKey Server url
    """
    url_config = {}

    # Parse the url
    o = urlparse(url)

    # Verify URL is a URL
    if not o.scheme:
        raise ExitCommandWithMessage("URL specified is not valid. Please enter valid URL")
    if not o.netloc:
        raise ExitCommandWithMessage("URL specified is not valid. Please enter valid URL")

    # Check if URL includes a context path
    test_url = o.scheme + '://' + o.netloc + '/project-getContainers.view'
    response = requests.get(test_url)
    if response.status_code == 200:
        context_path = None
    else:
        context_path = o.path.split('/')[1]
        test_url = o.scheme + '://' + o.netloc + '/' + context_path + '/project-getContainers.view'
        response = requests.get(test_url)
        if response.status_code != 200:
            raise ExitCommandWithMessage("URL specified is not running a LabKey Server. Please enter valid URL")

    # Determine if URL includes a container path
    # This assumes using new style LabKey URLs
    if context_path is None:
        if len(o.path.split('/')) < 3:
            # The url does not contain a container path
            url_config['container_path'] = None
        else:
            url_config['container_path'] = o.path.rsplit('/', 1)[0]
    else:
        urlpath = o.path.split('/', maxsplit=2)
        if len(urlpath) < 4:
            # The url does not contain a container path
            url_config['container_path'] = None
        else:
            url_config['container_path'] = urlpath[2].rsplit('/', 1).join('/')

    # Create output for function
    url_config['labkey_server'] = o.netloc
    url_config['context_path'] = context_path
    if o.scheme == 'http':
        url_config['use_ssl'] = False
        url_config['verify_ssl'] = False
        url_config['url'] = "http://" + o.netloc
    elif o.scheme == 'https':
        url_config['use_ssl'] = True
        url_config['verify_ssl'] = True
        url_config['url'] = "https://" + o.netloc
    else:
        raise ExitCommandWithMessage("URL specified is not valid. Please enter valid URL")

    return url_config

To use this function, all you need to do is

from labkey.utils import create_server_context
import urlparse

labkey_url = "https://www.labkey.org/home/Developer/issues/project-begin.view?"
# Validate LabKey Server URL break into component parts
url_config = read_url(labkey_url)
server_context = create_server_context(url_config['labkey_server'],
                                       url_config['container_path'],
                                       url_config['context_path'],
                                       use_ssl=url_config['use_ssl'],
                                       verify_ssl=url_config['verify_ssl']
                                       )
....


I hope this helps.

Brian

PS: I also have some code that can be used to access the server using username/password (via basic auth) instead of relying on a netrc file. I would be happy to share if anyone is interested.

view message
Setting Audit Detail Level
(2 responses) marcia hon 2018-09-17 10:40

Hello,

I created a custom table using modules.

How can I enable auditing of these custom tables.

Thanks,
Marcia

view message
Audit - Granularity
(2 responses) marcia hon 2018-09-12 10:32

Hi,

I would like to ask you if in the Audit Log, we can see what records/variables were added/updated/deleted?

I need to have this very fine granularity.

Thanks,
Marcia

view message
Editing more than one record at a time
(1 response) marcia hon 2018-09-10 09:24

Hello,

In my use-case, we would like the option whereby more than one record is updated at the same time.

Please let me know how this may be achieved as it is a crucial ability in our LabKey project.

Thanks,
Marcia

view message
Joining tables based on Subject Id and Visits/Sequence
(3 responses) Edward 2018-09-05 05:01

Hi,

I have a visit based study where there are multiple observations for a specific subject per visit. I have defined a sequence map (e.g., 1-10 sequence belongs to visit 1). Is there a way to create a data grid by joining the tables not based on visits sequence but by subject ids and any other column?

The reason is because if I try to join the two table where in one table I repeated visits and the second table has only one visit per subject, then the resulting grid has only visit joined and all the other rows are appended as if they belong to a different visit, which is not true.

In database terminology, I am talking about an outer merge of two tables based on subject id and visit ( not by subject id and visit sequence number).
Can anybody help here?

view message
Charts not displaying correctly
(3 responses) bront 2018-09-04 09:44

hi,

After upgrading to 18.2, I notice that charts aren't displaying correctly. Oddly, the data is is being displayed, but there are no graphics representing the bars or pieces of the pie charts (see attachment).

Here are the required files that worked prior to the upgrade:

"clientapi/ext4/Util.js",
"clientapi/ext4/data/Reader.js",
"clientapi/ext4/data/Proxy.js",
"clientapi/ext4/data/Store.js",

Here are the files I am requiring additionally in my efforts to fix the issue:

"vischart.min.js",
"vis/genericChart/genericChartHelper.js",
"vis/lib/crossfilter-1.3.11.min.js",
"vis/vis.min.js"

I am using the following to create the charts:

Ext4.create('Ext4.chart.Chart', { })

Am I missing a CSS file? Should I be using LABKEY.vis or LABKEY.chart? Something else?

Thanks for your help.

bront

 chart_issue.png 
view message
Tables - Change Values for Multiple Rows
(3 responses) marcia hon 2018-08-15 08:31

Hello,

I have a dataset/assay/list table.

There are multiple rows.

I would like to change the value for one of the columns for all of the rows.

I know that I can do this on a row-by-row basis using "edit".

However, is it possible to do this at the same time for multiple rows?

Thanks,
Marcia

view message
ExecutingSelector; bad SQL grammar []; nested exception is org.postgresql.util.PSQLException: ERROR: relation "exp.ObjectPropertiesView" does not exist Position: 714
(1 response) marcia hon 2018-08-10 11:04

Hello,

Another error... how can i solve this?

An unexpected error occurred
ExecutingSelector; bad SQL grammar []; nested exception is org.postgresql.util.PSQLException: ERROR: relation "exp.ObjectPropertiesView" does not exist Position: 714

org.springframework.jdbc.BadSqlGrammarException: ExecutingSelector; bad SQL grammar []; nested exception is org.postgresql.util.PSQLException: ERROR: relation "exp.ObjectPropertiesView" does not exist
Position: 714
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:231)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73)
at org.labkey.api.data.ExceptionFramework$1.translate(ExceptionFramework.java:37)
at org.labkey.api.data.ExceptionFramework$1.translate(ExceptionFramework.java:31)
at org.labkey.api.data.SqlExecutingSelector$ExecutingResultSetFactory.handleSqlException(SqlExecutingSelector.java:450)
at org.labkey.api.data.BaseSelector.handleResultSet(BaseSelector.java:280)
at org.labkey.api.data.BaseSelector.getArrayList(BaseSelector.java:102)
at org.labkey.api.data.BaseSelector.getArrayList(BaseSelector.java:97)
at org.labkey.api.exp.OntologyManager.getPropertyObjects(OntologyManager.java:609)
at org.labkey.api.reports.model.ReportPropsManager.getPropertyValue(ReportPropsManager.java:175)
at org.labkey.api.reports.model.ReportPropsManager.getProperties(ReportPropsManager.java:243)
at org.labkey.api.reports.report.ReportDescriptor.initProperties(ReportDescriptor.java:382)
at org.labkey.query.reports.ReportServiceImpl._getInstance(ReportServiceImpl.java:436)
at org.labkey.query.reports.DatabaseReportCache$ReportCollections.lambda$new$0(DatabaseReportCache.java:63)
at org.labkey.api.data.BaseSelector.lambda$forEach$3(BaseSelector.java:314)
at org.labkey.api.data.BaseSelector.lambda$forEachMap$1(BaseSelector.java:232)
at org.labkey.api.data.BaseSelector.handleResultSet(BaseSelector.java:263)
at org.labkey.api.data.BaseSelector.forEachMap(BaseSelector.java:226)
at org.labkey.api.data.BaseSelector.forEach(BaseSelector.java:316)
at org.labkey.api.data.BaseSelector.forEach(BaseSelector.java:295)
at org.labkey.query.reports.DatabaseReportCache$ReportCollections.<init>(DatabaseReportCache.java:62)
at org.labkey.query.reports.DatabaseReportCache$ReportCollections.<init>(DatabaseReportCache.java:47)
at org.labkey.query.reports.DatabaseReportCache.lambda$static$0(DatabaseReportCache.java:45)
at org.labkey.api.cache.BlockingCache.get(BlockingCache.java:144)
at org.labkey.api.cache.BlockingCache.get(BlockingCache.java:81)
at org.labkey.api.cache.BlockingCache.get(BlockingCache.java:203)
at org.labkey.query.reports.DatabaseReportCache.getReportsByReportKey(DatabaseReportCache.java:122)
at org.labkey.query.reports.ReportServiceImpl.getReports(ReportServiceImpl.java:551)
at org.labkey.api.reports.report.view.ReportUtil.getReportsIncludingInherited(ReportUtil.java:330)
at org.labkey.api.query.QueryView.addReportViews(QueryView.java:1650)
at org.labkey.api.query.QueryView.createReportButton(QueryView.java:1386)
at org.labkey.study.assay.query.AssayListPortalView.populateButtonBar(AssayListPortalView.java:59)
at org.labkey.api.query.QueryView.setupDataView(QueryView.java:2002)
at org.labkey.api.query.QueryView.createDataView(QueryView.java:1990)
at org.labkey.study.assay.query.AssayListQueryView.createDataView(AssayListQueryView.java:95)
at org.labkey.study.assay.query.AssayListPortalView.createDataView(AssayListPortalView.java:51)
at org.labkey.api.query.QueryView.renderDataRegion(QueryView.java:2101)
at org.labkey.api.query.QueryView.renderView(QueryView.java:1854)
at org.labkey.api.view.WebPartView.renderInternal(WebPartView.java:370)
at org.labkey.api.view.HttpView.render(HttpView.java:136)
at org.labkey.api.view.HttpView.render(HttpView.java:121)
at org.labkey.api.view.HttpView.include(HttpView.java:516)
at org.labkey.api.view.HttpView.include(HttpView.java:493)
at org.labkey.api.view.HttpView.include(HttpView.java:481)
at org.labkey.api.view.VBox.renderView(VBox.java:80)
at org.labkey.api.view.WebPartView.renderInternal(WebPartView.java:370)
at org.labkey.api.view.HttpView.render(HttpView.java:136)
at org.labkey.api.view.HttpView.render(HttpView.java:121)
at org.labkey.api.view.HttpView.include(HttpView.java:516)
at org.labkey.api.view.HttpView.include(HttpView.java:493)
at org.labkey.api.view.HttpView.include(HttpView.java:481)
at org.labkey.api.view.VBox.renderView(VBox.java:80)
at org.labkey.api.view.WebPartView.renderInternal(WebPartView.java:370)
at org.labkey.api.view.HttpView.render(HttpView.java:136)
at org.labkey.api.view.HttpView.render(HttpView.java:121)
at org.labkey.api.view.HttpView.include(HttpView.java:516)
at org.labkey.api.view.HttpView.include(HttpView.java:493)
at org.labkey.jsp.compiled.org.labkey.api.view.template.HomeTemplate_jsp._jspService(HomeTemplate_jsp.java:224)
at org.labkey.api.view.JspView.renderView(JspView.java:168)
at org.labkey.api.view.WebPartView.renderInternal(WebPartView.java:370)
at org.labkey.api.view.HttpView.render(HttpView.java:136)
at org.labkey.api.view.HttpView.render(HttpView.java:121)
at org.labkey.api.action.SpringActionController.renderInTemplate(SpringActionController.java:528)
at org.labkey.api.action.SpringActionController.handleRequest(SpringActionController.java:423)
at org.labkey.api.module.DefaultModule.dispatch(DefaultModule.java:1208)
at org.labkey.api.view.ViewServlet._service(ViewServlet.java:190)
at org.labkey.api.view.ViewServlet.service(ViewServlet.java:124)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.labkey.api.data.TransactionFilter.doFilter(TransactionFilter.java:38)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.labkey.core.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:118)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.labkey.api.module.ModuleLoader.doFilter(ModuleLoader.java:1144)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.labkey.api.security.AuthFilter.doFilter(AuthFilter.java:206)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:528)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1100)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:687)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1520)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1476)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.postgresql.util.PSQLException: ERROR: relation "exp.ObjectPropertiesView" does not exist
Position: 714
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2453)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2153)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:286)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:432)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:358)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:171)
at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:119)
at org.apache.tomcat.dbcp.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:82)
at org.apache.tomcat.dbcp.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:82)
at org.labkey.api.data.dialect.StatementWrapper.executeQuery(StatementWrapper.java:1157)
at org.labkey.api.data.SqlExecutingSelector$ExecutingResultSetFactory.executeQuery(SqlExecutingSelector.java:391)
at org.labkey.api.data.SqlExecutingSelector$ExecutingResultSetFactory.getResultSet(SqlExecutingSelector.java:331)
at org.labkey.api.data.BaseSelector.handleResultSet(BaseSelector.java:261)
... 95 more
DialectSQL = SELECT *
FROM (
SELECT
ObjectPropertiesView.ObjectId AS ObjectId,
ObjectPropertiesView.Container AS Container,
ObjectPropertiesView.ObjectURI AS ObjectURI,
ObjectPropertiesView.OwnerObjectId AS OwnerObjectId,
ObjectPropertiesView.name AS name,
ObjectPropertiesView.PropertyURI AS PropertyURI,
ObjectPropertiesView.RangeURI AS RangeURI,
ObjectPropertiesView.TypeTag AS TypeTag,
ObjectPropertiesView.FloatValue AS FloatValue,
ObjectPropertiesView.StringValue AS StringValue,
ObjectPropertiesView.DatetimeValue AS DatetimeValue,
ObjectPropertiesView.MvIndicator AS MvIndicator,
ObjectPropertiesView.PropertyId AS PropertyId,
ObjectPropertiesView.ConceptURI AS ConceptURI,
ObjectPropertiesView.Format AS Format
FROM exp."ObjectPropertiesView" ObjectPropertiesView ) x
WHERE (ObjectURI = ?) AND (Container = ?)

...

view message
ExecutingSelector; bad SQL grammar []; nested exception is org.postgresql.util.PSQLException: ERROR: relation "comm.Threads" does not exist Position: 681
(1 response) marcia hon 2018-08-10 10:54

Hello,

How is this error created? How can I fix it?

An unexpected error occurred
ExecutingSelector; bad SQL grammar []; nested exception is org.postgresql.util.PSQLException: ERROR: relation "comm.Threads" does not exist Position: 681

org.springframework.jdbc.BadSqlGrammarException: ExecutingSelector; bad SQL grammar []; nested exception is org.postgresql.util.PSQLException: ERROR: relation "comm.Threads" does not exist
Position: 681
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:231)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73)
at org.labkey.api.data.ExceptionFramework$1.translate(ExceptionFramework.java:37)
at org.labkey.api.data.ExceptionFramework$1.translate(ExceptionFramework.java:31)
at org.labkey.api.data.SqlExecutingSelector$ExecutingResultSetFactory.handleSqlException(SqlExecutingSelector.java:450)
at org.labkey.api.data.BaseSelector.handleResultSet(BaseSelector.java:280)
at org.labkey.api.data.BaseSelector.getArrayList(BaseSelector.java:102)
at org.labkey.api.data.BaseSelector.getArrayList(BaseSelector.java:97)
at org.labkey.announcements.model.AnnouncementManager.getAnnouncements(AnnouncementManager.java:115)
at org.labkey.announcements.AnnouncementsController$AnnouncementWebPart$MessagesBean.<init>(AnnouncementsController.java:2316)
at org.labkey.announcements.AnnouncementsController$AnnouncementWebPart$MessagesBean.<init>(AnnouncementsController.java:2306)
at org.labkey.announcements.AnnouncementsController$AnnouncementWebPart.<init>(AnnouncementsController.java:2255)
at org.labkey.announcements.AnnouncementsController$AnnouncementWebPart.<init>(AnnouncementsController.java:2286)
at org.labkey.announcements.AnnouncementsController$AnnoucementWebPartFactory.getWebPartView(AnnouncementsController.java:2348)
at org.labkey.api.view.Portal.getWebPartViewSafe(Portal.java:1208)
at org.labkey.api.view.Portal.populatePortalView(Portal.java:995)
at org.labkey.api.view.Portal.populatePortalView(Portal.java:959)
at org.labkey.core.portal.ProjectController$BeginAction.getView(ProjectController.java:327)
at org.labkey.core.portal.ProjectController$BeginAction.getView(ProjectController.java:280)
at org.labkey.api.action.SimpleViewAction.handleRequest(SimpleViewAction.java:80)
at org.labkey.api.action.BaseViewAction.handleRequest(BaseViewAction.java:177)
at org.labkey.api.action.SpringActionController.handleRequest(SpringActionController.java:415)
at org.labkey.api.module.DefaultModule.dispatch(DefaultModule.java:1208)
at org.labkey.api.view.ViewServlet._service(ViewServlet.java:190)
at org.labkey.api.view.ViewServlet.service(ViewServlet.java:124)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.labkey.api.data.TransactionFilter.doFilter(TransactionFilter.java:38)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.labkey.core.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:118)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.labkey.api.module.ModuleLoader.doFilter(ModuleLoader.java:1144)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.labkey.api.security.AuthFilter.doFilter(AuthFilter.java:206)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:528)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1100)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:687)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1520)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1476)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.postgresql.util.PSQLException: ERROR: relation "comm.Threads" does not exist
Position: 681
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2453)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2153)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:286)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:432)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:358)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:171)
at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:119)
at org.apache.tomcat.dbcp.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:82)
at org.apache.tomcat.dbcp.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:82)
at org.labkey.api.data.dialect.StatementWrapper.executeQuery(StatementWrapper.java:1157)
at org.labkey.api.data.SqlExecutingSelector$ExecutingResultSetFactory.executeQuery(SqlExecutingSelector.java:391)
at org.labkey.api.data.SqlExecutingSelector$ExecutingResultSetFactory.getResultSet(SqlExecutingSelector.java:331)
at org.labkey.api.data.BaseSelector.handleResultSet(BaseSelector.java:261)
... 53 more
DialectSQL = SELECT *
FROM (
SELECT
Threads.RowId AS RowId,
Threads.EntityId AS EntityId,
Threads.CreatedBy AS CreatedBy,
Threads.Created AS Created,
Threads.ResponseCreatedBy AS ResponseCreatedBy,
Threads.ResponseCreated AS ResponseCreated,
Threads.AssignedTo AS AssignedTo,
Threads.Container AS Container,
Threads.Title AS Title,
Threads.Body AS Body,
Threads.RendererType AS RendererType,
Threads.Expires AS Expires,
Threads.Status AS Status,
Threads.LatestId AS LatestId,
Threads.discussionsrcidentifier AS discussionsrcidentifier,
Threads.discussionsrcurl AS discussionsrcurl,
Threads.ResponseCount AS ResponseCount,
Threads.Modified AS Modified,
Threads.LastIndexed AS LastIndexed
FROM comm."Threads" Threads ) x
WHERE (Container = ?)
ORDER BY Created DESC
LIMIT 101

...

view message
Downloading Files from Labkey Server
(1 response) jin 2018-08-08 16:30

I'm wondering if there's a way to download files from Labkey using a shell script or something similar. Is it possible using the WebDav functionality of Labkey?

Thanks!

view message
Full text search over hard table
(3 responses) Will Holtz 2018-08-06 17:19

Is there a mechanism I can utilize in a file-based module to get a hard tabled indexed by the full-text search indexer?

thanks,
-Will

view message
RLabkey: makeFilter on non-existant column returns all values
(1 response) Will Holtz 2018-08-06 13:52

Hi,

I was surprised to see that the following R code returns the total number of users and not zero:

library(Rlabkey)
nrow(labkey.selectRows(
baseUrl="http://localhost:8080/labkey/",
folderPath="/",
schemaName="core",
queryName="users",
colFilter=makeFilter(c("foo", "EQUAL", "bar")),
) )

Perhaps that is an 'undefined' operation and therefore this output is valid. I would find it more useful to get zero back as that is more likely to break my code and make me realize I made a typo in my column name. This is with LabKey v18.2 and Rlabkey v2.2.2.

thanks,
-Will

view message
assayFileUpload: larger file from JS variable
(1 response) Tim Holland 2018-08-01 08:12

Hi,

I'm attempting to use LABKEY.Ajax.request to create an assay file with assayFileUpload using the contents of a Javascript variable, but when the fileContents parameter gets too big (even just on the order of a few hundred lines) the request fails. It uses the method described in the documentation here: https://www.labkey.org/download/clientapi_docs/javascript-api/symbols/LABKEY.Exp.Data.html, with something like the following:

LABKEY.Ajax.request({
    url: LABKEY.ActionURL.buildURL("assay", "assayFileUpload", LABKEY.ActionURL.getContainer()),
    params: { protocolId: assay.id, fileName: name, fileContent: lines },
    success: function(response) {
        // ...do something...
    },
    failure: formFailed
});

Is there a recommended way to create a larger file from the contents of a Javascript variable that gets around this limitation? I'd prefer not to have to edit the Tomcat configuration files to do this.

view message
Can't declare foreign key relationship in module schemas/<schema>.xml metadata
(1 response) Will Holtz 2018-07-30 18:47

Hi,

I made a hard table that contains a foreign key to a Labkey userid (beyond the standard userid fields of createdby, modifiedby, owner). In the module schemas/<schema>.xml metadata I declared the foreign key. In my grid views, I could not get the userid lookup to occur; the integer value was displayed, not the displayname field. I know there are a lot of restrictions on access to core.users and related tables and I thought I was just not getting some of that right. However, I eventually realized that by declaring the foreign key in the module queries/<schema>/<query>.query.xml metadata, I was able to get the lookup to work. It is unclear to me if there is some reason why the foreign key declaration in the schema/<schema>.xml file did not work or if this behavior is unexpected.

I made a minimal reproduction that is contained in the attached module file. In this module, two identical hard tables are created. UserMetadata.SchemaMetadata has the foreign key declared in schemas/UserMetadata.xml. UserMetadata.QueryMetadata has the foreign key declared in queries/UserMetadata/QueryMetadata.query.xml. In both tables, "associateduser" is the userid field I am trying to get the foreign key metadata applied to such that the lookup of displayname occurs. In the Labkey query browser, the associateduser field in UserMetadata.SchemaMetadata does not show a lookup column, while the same field in the UserMetadata.QueryMetadata table has "core.Users.UserId (DisplayName)" as the Lookup column.

I did my testing on Labkey v18.2, Postgres v10.4, Tomcat v8.5.24.

thanks,
-Will

 UserMetadata-18.2.module 
view message
Limited number of records in executeSql
(1 response) zoya_pasha 2018-07-20 02:00
Hello,

I am using JavaScript API of Labkey 17.1. On using the function LABKEY.Query.executeSql, I am able to fetch records up to 100000. I checked the documentation. The default value for the same is 'All rows'. However, the maximum number of rows I am able to get in one transaction is limited. Please help!

Regards
Zoya
view message
TeamCity 2018.1 Upgrade
(1 response) Trey 2018-07-16 14:02

We plan on upgrading TeamCity to 2018.1 later this week.
In preparation, we may need to take it offline briefly over the next day or two. We will send out a warning an hour before any such outage. They should last no more than 15 minutes; in-progress builds will be unaffected and the server should pick up right where it left off once it comes back up.

The upgrade itself may take the better part of a day. This thread will be updated at least 24 hours before we start the upgrade process.

view message
Gridviews per User
(1 response) marcia hon 2018-07-03 12:17

Hello,

I have a big table.

Now, I wish to filter it so that only user_x would be able to view the table filtered on x and nothing else.

For example, I have a table :

Name Department Date
Marcia English Nov 21, 2018
Bobby French Aug 12, 2017
John Portuguese Oct 30, 2016

So, I want Marcia to only see data filtered on "English". With Bobby, "French" and with John "Portuguese".

Is this possible and how do you do this?

Thanks,
Marcia

view message
Data Transforms
(1 response) marcia hon 2018-06-28 10:28

Hello,

I have different users who use different parts of a project.

Now, there are around 6 data transformations.

I would like to refine this so that a certain individual will only see 1 data transform.

Is this possible?

Thanks,
Marcia

view message
Rlabkey calls from Transformation script result in 401 - User does not have permission to perform this operation
(1 response) Will Holtz 2018-06-27 16:42

I want to share a workaround to a problem I had. I'm not expecting anyone to follow up on this as I don't know how to reproduce the issue.

I have several file-based modules that contain assays with transformation scripts written in R. These transformation scripts use Rlabkey to read and write data from the Labkey instance. Today, these modules started hanging during the transformation scripts. It appears the Rlabkey calls back to the Labkey instance were failing with a 401 error and the error message ' User does not have permission to perform this operation.' This would even happen when performing the upload as a Labkey admin user. I created a test module that contained a minimal transformation script:

library('Rlabkey')
${rLabkeySessionId}
print(labkey.selectRows(
    baseUrl = '${baseServerURL}',
    folderPath = '/',
    schemaName = 'core',
    queryName = 'users',
))

On my development instance on my laptop, this transformation script worked fine, but on my production server it gave the 401 error. After a few hours of trying to figure out what was going on, I restarted my tomcat instance and everything started working again. Reloading the application within tomcat did not help -- only a full tomcat stop and start worked. So, if you are in a similar situation, try a tomcat restart.

I'm running Labkey v18.1, Tomcat v8.5.30, R v3.4.4, Rlabkey v2.2.1, R package httr v1.3.1 and R package rjson v0.2.19 on Ubuntu v18.04.

-Will

view message
Use trigger script to provide info message
(1 response) Sev 2018-06-25 12:40

Hello

We would like to give users an informational message following a dataset row insert that meets certain criteria. We have trigger scripts setup to check the criteria but it seems the only way to provide messages to the user is via the errors object, which would cancel the insert.

Is there a way we can set a message after an insert into a dataset that wont result in the transaction being cancelled?

Thanks

Sev

view message
Transformation Jobs
(1 response) marcia hon 2018-06-21 08:13

Hello,

I have several Transformation Jobs.

Is it possible to have "Data Transforms" view only show the Transformations we want?

For example,

Let's say I have transformation jobs: T1, T2, T3, T4, T5

I would like the "Data Transforms" webpart to only show : T3, T5

Thanks,
Marcia

view message
File Repository Module write location, etc.
(3 responses) Georgia Mayfield 2018-06-20 12:05

Hi all,

My team is considering using the File Repository module for file upload/storage and has some questions on its functionality/implementation.

First off, where does File Repository actually write the uploaded files (by default)? I'm unable to track uploaded test files on either the machine or in the database. Is there an API into the Webdav repository? Also, is there any sort of built-in backup for File Repository?

Is there's no backup in File Repository itself, then we would need to map the write space to a network drive, such as an RDS repository. What would you estimate to be the complexity/requirements of that?

Additionally, we would be interested in keeping metadata about the uploaded files and clickable links to the files in a database table to retain out current functionality. What sort of metadata capture features like this exist for File Repository, if any? Would it be possible to cap file size, if needed?

System Info

  • OS: rhel 6
  • Version: 17.3

Thanks,
Georgia Mayfield
OHSU Computational Biology

view message
LImit to insertrows query
(9 responses) WayneH 2018-06-20 07:01

Hello all,

we employ a script on our labkey server v16.3 (prof+ edition) that inserts a data array of 10k+ entries via the insertrows api call and it is failing.. it has no issues with smaller chunks of data so it appears that we have crossed a size limit.. a fix is kinda obvious but wanted to confirm with you what those limits are and if chunking things would be the solution or you may have an alternate approach you could recommend.

thanks,
WayneH

view message
Gradle Plugins Version 1.3
(1 response) Susan Hert 2018-06-18 16:55

With r58720 in trunk, the version of gradlePlugins has been updated to 1.3. As always, the full list of changes in this version are available in the README for the gradlePlugin repository. There are a couple of changes here that I wanted to call your attention to, one of which requires action from you for better success in building.

For the impatient, the summary is:

  • The build process will now download and use particular versions of npm and node. To avoid conflicts with node module versions, you need to remove any existing node_modules directories in the LabKey modules' enlistment directories before doing a build with the 1.3 gradlePlugin version. (If using npm from the command line, you should also update your path to reference the downloaded versions.)
  • Local builds will now fail if a conflict in version numbers is detected between two builds of a module (but you can change this behavior if you so desire).

Node and NPM

The build process will download the versions of npm and node that are specified by the properties npmVersion and nodeVersion, respectively, which are defined in the root-level gradle.properties file of the LabKey enlistment.

To avoid potential conflicts with node module versioning, you should remove any existing node_modules directories from your LabKey modules' directories before doing your first build with the 1.3 version. The 1.3 version of gradlePlugin provides a task, cleanNodeModules, that does just this. You can invoke this task at the root level of your enlistment and it will execute the task for all LabKey modules that have a package.json file. Alternatively, you can manually remove the node_module directory for each LabKey module that has one.

When running npm from the command line, you will want to be sure to use the version of npm that the gradle build uses. You can reference the versions for the core module in the <LABKEY_ROOT>/build/modules/core/.node/ directory.

On non-Windows platforms, the deployApp command creates symlinks for both node and npm in the directory .node at the root level of your enlistment:

trunk$ ls -la .node/
total 0
drwxr-xr-x   4 developer  staff   128 Jun 14 14:11 .
drwxr-xr-x  38 developer  staff  1216 Jun 18 10:36 ..
lrwxr-xr-x   1 developer  staff    90 Jun 14 14:11 node -> /Users/developer/Development/labkey/trunk/build/modules/core/.node/node-v8.11.3-darwin-x64
lrwxr-xr-x   1 developer  staff    77 Jun 14 13:58 npm -> /Users/developer/Development/labkey/trunk/build/modules/core/.node/npm-v5.6.0

If on Windows, you can create the symbolic links in the <LABKEY_ROOT>/.node directory manually to point to the node and npm versions in the core module's .node directory, as shown above. You should then put <LABKEY_ROOT>/.node/node/bin and <LABKEY_ROOT>/.node/npm/bin in your path.

If not using npm on the command line, the symbolic links and path changes are not necessary. (Note that the <LABKEY_ROOT>/.node directory won't exist until you run a deployApp command after updating to version 1.3.)

Local Build Version Discrepancies

With the 1.3 version, a feature that was incubating in the last release has now come out of incubation with a slight behavior change. You may have noticed if you did a build after someone updated a version of an external dependency that you would get a warning message something like this about version conflicts that would be produced by the build.

WARNING: Artifact versioning problem(s) in directory /Users/developer/Development/labkey/trunk/build/modules/api/explodedModule/lib:
  Conflicting version of commons-compress jar file (1.14 in directory vs. 1.16.1 from build).
  Conflicting version of objenesis jar file (1.0 in directory vs. 2.6 from build).

Now your build will fail if such version conflicts are detected (because the build would not be in a good state if both versions were included). If you need to, you can change that behavior by setting a value for the versionConflictAction parameter, as described in the Version Conflicts in Local Builds documentation.

view message
How to make a table not be deleteable
(1 response) marcia hon 2018-06-13 08:50

Hello,

I have created a table via module-etl.

Is there a way to force the table to not allow records to be deleted?

Thanks,
Marcia

view message
Stored Procedure and Lists Dropdown List
(1 response) marcia hon 2018-06-13 07:26

Hello,

I have a list that I wish to treat like a lookup table.

I have a foreign key for it via a table i created in etl module.

This linkage works, however, visually, there is no dropdownlist that informs the user of what values are appropriate.

Please could you let me know how to make this dropdownlist appear?

Thanks,
Marcia

view message
Grid button bars and the new UI
(1 response) Will Holtz 2018-06-08 10:53

In a file-based module, Is there a recommended way to add custom buttons with icons instead of text to a grid button bar? I'd like to get my custom buttons to match the look of the standard buttons, but don't see an obvious way of doing this short of hacking around in the DOM.

thanks,
-Will

view message
R Report Error
(3 responses) marcia hon 2018-06-08 10:19

Hello,

I get the following error when trying to run an R Report:

Error executing command
javax.script.ScriptException: javax.script.ScriptException: An error occurred when running the script 'script.R', exit code: 1).

My script is the following:

knitr::opts_chunk$set(echo=FALSE, cache=FALSE, comment=FALSE, comments = "", warning=FALSE, eval = TRUE, encoding = "native.enc")
#knitr::opts_chunk$get();
knitr::pandoc('script.Rmd', format='html')

currentdir <- getwd()
assign("html_filename", paste(currentdir, "/script.html", sep=""))
assign("out_filename", paste("/nip/NRG_Samples/@files/Reports/GenotypingResults_CNR1rs2023239_DRD3rs6280/Report_GenotypingResults_CNR1rs2023239_DRD3rs6280_", format(Sys.time(), "%Y-%m-%d_%H:%M:%S"), ".html", sep=""))

#list.files(path = ".", pattern = NULL, all.files = FALSE,
#           full.names = FALSE, recursive = FALSE,
#           ignore.case = FALSE, include.dirs = FALSE, no.. = FALSE)

file.copy(from = html_filename, to = out_filename)

system("rm script.* *.tsv")
#system("rm cache/*.*")
#list.files(path = ".", pattern = NULL, all.files = FALSE,
#           full.names = FALSE, recursive = FALSE,
#           ignore.case = FALSE, include.dirs = FALSE, no.. = FALSE)

<html>...</html>

I look forward to your suggestions.

Thanks,
Marcia

view message
LABKEY.QueryWebPart.standardButtons documentation is out of date
(2 responses) Will Holtz 2018-06-08 10:18

It appears that some of the LABKEY.QueryWebPart.standardButtons no longer work on v18.1.

<div id="ExampleDiv" />
<script>
new LABKEY.QueryWebPart({
        renderTo: 'ExampleDiv',
        schemaName: 'core',
        queryName: 'users',
        containerPath: '/',
        buttonBar: {
            items:[
              LABKEY.QueryWebPart.standardButtons.insertNew,
              LABKEY.QueryWebPart.standardButtons.pageSize
            ]
        },
 });
<script/>

Results in a message at the top of the grid, "WARNING: button bar configuration contains reference to buttons that don't exist. Invalid original text: insert new, paging."

-Will

view message
Error in clicking "EDIT" or "DETAILS" of a ETL created table
(1 response) marcia hon 2018-06-06 08:19

I created a table via ETL module and I populate it via a StoredProcedure module.

However, when I try to Edit / see Details of the records, I get this error:

An unexpected error occurred
ExecutingSelector; uncategorized SQLException for SQL []; SQL state [XX000]; error code [0]; ERROR: failed to find conversion function from unknown to text; nested exception is org.postgresql.util.PSQLException: ERROR: failed to find conversion function from unknown to text

I look forward to your response.

Thanks,
Marcia

view message
Data Grids and updating the URL to reflect current state
(4 responses) Will Holtz 2018-06-04 23:23

executeQuery.view does an excellent job of updating the page URL to keep track of the current filters, sorting and paging, such that if a user follows a link from the data grid and then uses the back button, the page is the same as when they left it. However, executeQuery.view isn't easy to extend without writing Java code. I frequently use javascript QueryWebParts so that I can easily customize and extend data grids, but my users get annoyed that these QueryWebPart data grids do not nicely retain their state like executeQuery.view does.

Is there some simple way to get QueryWebParts to update the URL to capture the current state of the grid? I understand there is the complexity of needing to deal with multiple QueryWebParts on a single page, but that doesn't seem like a show stopper. I likely can build this out using LABKEY.DataRegion events, but I can't imagine I'm the first person with this need and I'd rather not reinvent the wheel.

thanks,
-Will

view message
Deleting record in Assay from Stored Procedure
(1 response) marcia hon 2018-06-04 09:50

Hello,

I have an ETL Stored Procedure that gets data from an Assay.

Now, I would like that once this data has been copied to another table, that the data is erased in the Assay.

How do I do this in the Stored Procedure?

Thanks,
Marcia

view message
Filtering Datasets by row
(2 responses) WayneH 2018-06-01 13:17

Good afternoon,

I just wanted to know if there was a way to filter by row.. For example what we are trying to do is filter datasets to show rows that are not blank rather than applying a notnull filter to each field... But apply a global filter by row..

thanks.

Wayne H

view message
users groups roles for all projects in a single table
(3 responses) celikalper84 2018-05-31 10:27

Hi,

We have a platform that uses labkey community edition among other tools. Each of these tools have their own access regulations user roles etc and we want to combine them into a single location for tracking and auditing purposes.

I have been playing with the core schema of labkey and seems like a combination of principals, members, roleassignments, containers and usersdata should give me what I need. The information I'm trying to extract is as follows:

username, name, last name, email, group, groupid, role, container (path included if possible).

It seems like groups and subgroups are also treated like users, and the column names of these tables are not immediately intuitive to me.
I know that we can get this from view in the GUI but what we want to achieve is a daemon that can query this at certain intervals (every night or twice a day) in an automated fashion.

Is there a SQL query that can achieve this goal or at the very least where can I find information about how these tables are related to one another.

Thanks
Alper

view message
How to make custom tables' rows editable
(1 response) marcia hon 2018-05-30 05:52
Hello,

I created a module that consists of tables with the following definitions:

CREATE TABLE nrg01cmh_samples.Samples_Template (
    SeriesID VARCHAR(50),
    CollaboratorID VARCHAR(50),
    Sex VARCHAR(1),
    DateSampleDrawn DATE,
    DateSampleReceived DATE
);
---
   <ns:table tableName="Samples_Template" tableDbType="TABLE">
    <ns:columns>
      <ns:column columnName="SeriesID" />
      <ns:column columnName="CollaboratorID"/>
      <ns:column columnName="Sex"/>
      <ns:column columnName="DateSampleDrawn"/>
      <ns:column columnName="DateSampleReceived"/>
    </ns:columns>
   </ns:table>

However, when I access these tables in Labkey, I am not able to edit the rows as there is not "Edit" link.

The ability to edit these values is very important to our project.

Please let me know how to correct this issue.

Thanks,
Marcia
view message
Subversion migration complete
(1 response) jeckels 2018-05-23 20:01

The migration of our SVN server from hedgehog.fhcrc.org to svn.mgt.labkey.host is now complete.

As previously announced, the hedgehog.fhcrc.org server is now in read-only mode and it is not mirroring updates that are written to the new server, so you will need to relocate any enlistments you currently have. This is done with the 'svn relocate' command from the root of each enlistment.

For example, at the root of your trunk enlistment, you would use the following command:

svn relocate https://svn.mgt.labkey.host/stedi/trunk

And for an 18.1 enlistment, the command would be:

svn relocate https://svn.mgt.labkey.host/stedi/branches/release18.1

Your svn credentials will stay the same and cpas is still a read-only user on the new server. Note that you may need to include the --username argument to svn to make it use the correct account name. Patterns for the names have varied over the years, but an example would be:

svn relocate https://svn.mgt.labkey.host/stedi/trunk --username jeckels@labkey.com

Thanks,

Josh

view message
Multiple demographics datasets
(5 responses) Sev 2018-05-22 10:38

Hello

Our users have requested that we hide demographics data from all but a small set of admin users for our LabKey instance. To strike a balance between setting up this level of security and still being able to use datasets that refer back to the participantid in the Demographics table, we have created a separate AdminDemographics dataset to supplement the Demographics dataset. AdminDemographics contains PHI such as MRN, Names, Birth dates, etc and the Demographics dataset contains only the participant ID column. We keep these in sync with triggers.

We created the AdminDemographics dataset via:
CREATE TABLE LIKE studydataset.demographics INCLUDING ALL
We registered it with LabKey as a dataset with the additional key column field set to the _key column in the dataset, marked as system managed. When we rolled this out, however, LabKey had assigned the MRN column to this additional key and marked it as System Managed, see attached screenshot.

The impact of this was that MRN was not allowed to be set by users when adding new records, breaking the ability to add new patients to the system. We were able to correct this by manually resetting the additional key column back to none. I also noticed if I select the 'Demographic Data' check box the MRN is settable on insert.

My questions:

  1. Why was MRN assigned to a system managed additional key column? Is 'MRN' some kind of reserved column name that gets auto assigned? The _key column exists distinct of the MRN column in the dataset.
  2. Besides enforcing zero or 1 row per participant as described here, what impact does checking the 'Demographic Data' option have on the system? Apart from making MRN settable when adding new records we want to know what else is going on so we can test thoroughly.

EDIT: I attached csv files for the admindemographics entries in the study.dataset and exp.domaindescriptor tables in case that is helpful

Thanks!

Sev

 Screen Shot 2018-05-22 at 10.22.11 AM.png  exp.domaindescriptor_entry.csv  study.dataset_entry.csv 
view message
Rlabkey: labkey.selectRows with colSelect=c("*") gives error
(4 responses) Will Holtz 2018-05-21 14:36

Hello,

The Rlabkey documentation for the colSelect parameter of labkey.selectRows states, "(optional) a vector of strings specifying which columns of a dataset or view to import. The wildcard character ("*") may also be used here to get all columns including those not in the default view."

On LabKey v18.1:

 labkey.selectRows(
    baseUrl = 'http://localhost:8080/labkey/',
    folderPath = '/',
    schemaName = 'core',
    queryName = 'users',
    colSelect = c("*")
  )

returns:

Error in makeDF(mydata, colSelect, showHidden, colNameOpt) : 
  The column "*" specified in the colSelect variable does not exist in the result set. Be sure you are using the column name for colNameOpt="fieldname" and the column label for colNameOpt="caption". See the documentation for more details.
Called from: makeDF(mydata, colSelect, showHidden, colNameOpt)

This same call without the colSelect parameter defined returns the expected data frame populated with user attributes. Am I messing something up or is this a bug?

thanks,

-Will

view message
Subversion server migration scheduled - Wednesday May 23, 7PM PDT
jeckels 2018-05-21 10:52
As previously [communicated on this forum](https://www.labkey.org/home/Support/Developer%20Forum/announcements-thread.view?rowId=17652), we are switching from our long-standing Subversion server to a new system.

The cutover has now been scheduled for 7PM PDT on Wednesday May 23.

The new server is [https://svn.mgt.labkey.host/](https://svn.mgt.labkey.host/) and is actively mirroring the existing server, hedgehog. It retains full history from the previous repository. TeamCity is already pulling from the new server.

At the moment the new server is a read-only copy, and allows connections using the generic cpas/cpas username/password combination. At 7PM on Wednesday, the previous server will be flipped to be read-only, and accounts will be enabled on the new server. Existing credentials will be retained, so you should continue using whatever account you have been using.

The previous server will continue running for a few weeks in case any problems are found, but will be decommissioned soon.

After the cutover, you will be able to use SVN's "relocate" command to point an existing enlistment to the new server. It takes a few minutes and will retain any local edits you have in progress. More detail will follow at the time of migration.

Thanks,

Josh


Edit: Migration Complete. Please refer to the following post for instructions on using the new SVN repository:

https://www.labkey.org/home/Support/Developer%20Forum/announcements-thread.view?rowId=17762
view message
Intellij 2017 and Gradle build?
(2 responses) Ben Bimber 2018-05-21 09:18
I recently set up a new dev machine and it's been a while since I did much java development, but I'm finding builds/hot swapping to be painfully slow. I am using the latest Intellij, 2017.3. I assume this means I must be doing something out of sync with how most LabKey devs have this set up. I have a couple questions:

1) Are many devs using Intellij 2017 or still 2016?

2) I noticed Intellij and command line gradle seemed to be outputting to different directories (intellij was using paths like server\api\out, I think). This might relate to this issue:

https://youtrack.jetbrains.com/issue/IDEA-175172

Based on what was suggested there, I enabled:

File | Settings | Build, Execution, Deployment | Build Tools | Gradle | Runner => Delegate IDE build/run actions to gradle.

Which made a major difference on build time. However, I dont think classpath is set correctly since when I try to build the project through intellij, the GWT classes give errors. Is this a familiar issue for LK devs, and/or is there something else I should be looking into?

Thanks in advance.
view message
Using ExtJS to create an editable grid in a wiki page
(1 response) olnerdybastid 2018-05-10 14:55

I am looking to build an interface for our users to view & edit a table of sample set properties. Based on the docs I see here, it looks like the LabKey-recommended solution is to use ExtJS to achieve this. I tried to create a quick proof-of-concept example to see how this might work building a very simple interface on a wiki page, but the rendering of the ExtJS grid panel isn't remotely usable in its current form (see attached screenshot). I'm assuming the issue is easily solved by loading an additional stylesheet in my wiki, but I've yet to find one bundled with LabKey that properly renders my ExtJS grids. Is there a step that I'm missing in order to create usable ExtJS components in LabKey?

Below is the code I used to make this example:

<head>
<script type='text/javascript' src='https://mylkserver.brown.edu/labkey/ext-4.2.1/ext-all.js'></script>
<script type='text/javascript'>

LABKEY.requiresExt4ClientAPI(function(){

  var lkstore = Ext.create('LABKEY.ext4.data.Store', {
      containerPath: '/NDDdb',
      schemaName: 'assay.General.NanoDrop QC',
      queryName: 'NanoDropQCSummary',
      parameters: {'runID': 122},
      autoLoad: true
    });

  var panel = Ext.create('Ext.grid.Panel', {
      store: lkstore,
      layout: {
        type: 'vbox',
        align: 'stretch',
        padding: 5
      },
      columns: [{
          id: 'sID',
          header: 'P number',
          dataIndex: 'sID',
          flex: 1,
          editor: {
              allowBlank: false
          }
      }, {
            xtype: 'checkcolumn',
            header: 'QC status',
            dataIndex: 'QCstat',
            width: 60,
            editor: {
                xtype: 'checkbox',
                cls: 'x-grid-checkheader-editor'
            }
        }, {
          header: '# of reads',
          dataIndex: 'nReads',
          width: 20,
          editor: {
              xtype: 'combobox',
              typeAhead: true,
              triggerAction: 'all',
              selectOnTab: true,
              lazyRender: true,
              listClass: 'x-combo-list-small'
          }
      }],
      selModel: {
          selType: 'cellmodel'
      },
      renderTo: 'QCPanel',
      width: 600,
      height: 300,
      title: 'View QC data',
      frame: true
  });
})

</script>
</head>
<body>
  <div id = "QCPanel" /></form>
</body>

 Screen Shot 2018-05-10 at 5.37.14 PM.png 
view message
Automatic Assay Imports
(1 response) slatour 2018-05-09 13:06

Hi,

I am currently required to upload ~500 files and import them into assay. This is tedious for a couple of reasons.

  1. There is no "Select all" button within the repository so I must manually select each file and then select "Import Data" button from the File Repository.

  2. After (1) is completed I still need to click Save and Next Run for each item I want to bring into the Assay

Is there a programmatic way to do this? I have tried using both Python and R APIs with no avail. Python only allows one row updated per call which is not helpful considering each of the 500 files has ~ 1000 rows.

Thanks in advance for any feedback,

  • Sara
view message
Importing updated bulk data
(1 response) harris 2018-05-09 12:21

Hello,
I am using the import bulk data functionality to load data and the initial data load works out nicely. I am looking into operationalizing this and have a couple of questions.

  1. If I want to update a full set of data (i.e., overwrite the existing dataset), the import bulk data option does not allow me to do this. I've only been able to import new rows. Is there a better way to refresh the entire dataset without having to delete first and then re-import?
  2. Does LabKey have a way to keep track of study data snapshot versions? For Phase 1 dose escalation studies, we need to do this weekly and need to refer to data snapshots from previous weeks (i.e., I want to see the data we used for last week's dose escalation decision.).
 Screen Shot 2018-05-09 at 12.11.13 PM.png 
view message
Managing Visits: Visit Import Mapping
(1 response) harris 2018-05-09 12:02

Hello,
I imported custom visit mapping data, per the documentation instructions. It was my understanding that the Visit labels should be updated based on this custom mapping, but that does not seem to be the case. See attached snapshots.

  • Is there something I am missing to allow the visit labels per the custom mapping?
  • Is there a way to populate the visit labels without having to do this manually?
 Screen Shot 2018-05-09 at 11.59.05 AM.png  Screen Shot 2018-05-09 at 11.59.57 AM.png  Screen Shot 2018-05-09 at 11.58.01 AM.png 
view message
Query Selector Checkboxes on JOINED Query
(2 responses) simon buckner 2018-05-09 07:48
Hello,

I came across this post: https://www.labkey.org/home/Support/Developer%20Forum/announcements-thread.view?entityId=0af653dc-8023-1030-aaa0-987439a6474a

...which indicates that selectors are not shown for queries that use JOIN statements. However, it is fives years old, so I figured I would make a new thread.

We have a query view, and we would like to be able to select the rows. This query includes a JOIN statement, and as a result, does not seem to have a primary key. I thought this was because the joined queries might return duplicate primary keys, but it looks like the presence of a JOIN statement at all hides the selectors. Is this the case, and if so, is there a workaround at this time?

Thanks,

Simon
view message
Missing domain kind?
(1 response) eva pujadas 2018-05-07 23:11

Dear LabKey team,

We are running different instances of LabKey server, in version 18.1, and when performing a database check or validation, we always get the error that for a certain database table "audit.XXXXXX_queryloggingauditdomain" its domain kind is not found. E.g.

03 May 2018 09:16:35,236 ERROR: Could not find a domain kind for audit.c3d268_queryloggingauditdomain

Do you know the reason? Is there something to worry about?

Thanks and best,
Eva

view message
Upcoming Subversion server migration
jeckels 2018-05-01 19:19

Later this month, we will be migrating from our long-standing Subversion server, hedgehog.fhcrc.org, to a brand new server.

For automated systems like TeamCity, our automated build and test system, we will be doing a phased migration. We will start synchronizing the new server with the existing server, in a read-only capacity, that lets systems pull from it.

A little later, we will make the new server the system of record, and flip the original server to be a read-only system until it is completely retired. We will be migrating all branches, and retain the full source code history.

Stay tuned for more information, including target dates and instructions for how to migrate any existing enlistments you may have locally. We expect this should be a fairly painless operation for developers.

Thanks,
Josh

view message
Missing Values
(3 responses) harris 2018-05-01 17:24

Hi,
I'm trying to import data in an Assay folder. The file I wish to import has some missing data represented as N/A. For example, a data field called Area is Number(double), but does not allow me to upload the file if there are missing data. I tried following the documentation for Manage Missing Value Indicators, but still doesn't allow me to upload anything with N/A. Perhaps I have misunderstood the Missing Value Indicator functionality. Is there a way to translate specific values to missing so that data can be imported?

view message
Additional Keys
(8 responses) harris 2018-04-30 14:33

The Edit Dataset Properties for the Study folder allows one additional key column. How can we add more than one additional key column? I'm currently using Creating Multiple Dataset Definitions from a TSV File. Is it possible to specify additional keys in this definition file?

view message
upload file sanity check and verification in Labkey
(3 responses) WayneH 2018-04-30 05:43
Good morning all,

We want to verify that a file correctly matches a specific naming convention upon completion of upload.. Via the files web part or javascript. Do you have any insight as to how we may accomplish this?


Thanks,

Wayne H
view message
Querying for median value in LabKey?
(2 responses) olnerdybastid 2018-04-25 08:47

I am writing a query to summarize some assay results with some very basic statistics (std dev, average, and median). The first two are already taken care of using standard LabKey SQL, but calculating median looks to be a bit more difficult unless I'm overlooking something

Our LabKey instance is deployed over Postgres, and I've seen multiple examples of how to do this natively in Postgres (one example here: https://www.periscopedata.com/blog/medians-in-sql). But I don't see anything analogous to to Postgres' ROW_NUMBER() in LabKey SQL, or any way to select an element by its position in a subselect. Is there a way to calculate median in a LabKey SQL query, or has anyone found an acceptable workaround to achieve this?

view message
Custom Schema and Table
(3 responses) marcia hon 2018-04-19 09:20

Hello,

I created a custom schema and table in Labkey via ETL script.

Now, when I am at Labkey GUI and open this table, the option to "delete" is unavailable.

Please is there a way to allow for "delete".

Thanks,
Marcia

view message
Transformation Scripts
(1 response) marcia hon 2018-04-19 08:08

Hello,

I have an assay: assay.general.nrg_assay .

How do I reference this assay in the transformation scripts?

In the ETL structure, where do I put this scirpt?

Is it supposed to be ../queries/assay/general/nrg_assay.sql?

Thanks,
Marcia

view message
Transformation Scripts
(1 response) marcia hon 2018-04-18 12:42

I would like to the following script to be the transformation script:

SELECT
Genotyping_Original."participant id"
FROM study.Genotyping_Original as Genotyping_Original
WHERE NOT EXISTS
(
SELECT *
FROM genotype_etl.genotyping_original as GenoETL
WHERE
Genotyping_Original."participant id" = GenoETL."participant id"
)

However, it is always not giving me the correct results.

I think it's because there are two queries being referenced: study.Genotpying_Original and genotype_etl.genotyping_original . So it seems that it is not able to get information from genotype_etl.genotyping_original .

genotype_etl.genotyping_original is a schema and table that i create in the same transform.module package.

I look forward to your expertise.

Thanks,
Marcia

view message
Error: removing module and delete schema
(1 response) marcia hon 2018-04-17 07:36

At Admin > Site > Admin Console,

I'm trying to delete a custom module/schema.

I go to the screen that has the following and click "OK":

"
Are you sure you want to remove the "genotype_etl" module and delete all data in the "genotype_etl" schema? This operation may render the server unusable and cannot be undone!

Deleting modules on a running server could leave it in an unpredictable state; be sure to restart your server.
"

Unfortunately, it runs for a long time and times out. The module is not removed and the schema is not deleted.

Please let me know how to fix this.

Thanks,
Marcia

view message
Dataset and ETL
(1 response) marcia hon 2018-04-12 07:35

Hello,

I would like the following workflow:

  1. Assay data is uploaded and modified
  2. Copy assay to a dataset
  3. Via ETL, pass this data to another table (created with a module).
  4. The stored procedure determines into which tables data goes to.

How can I make sure that duplicates don't happen between steps 2 and 3?

Thanks,
Marcia

view message
Handling deletes on ETL where the source is a LK dataset?
(3 responses) Ben Bimber 2018-04-11 13:28

The ETL allows you to have a RemoteQuery source, such as a dataset on another server. If the source table is large and relatively stagnant, doing an incremental merge (based on the modified field) is in theory nice. However, I need to deal with the potential of deletes from the source table. In our case, the source has auditing, so I assume I could use the audit tables to try to figure that out (although queries against this are painfully slow). Is there an example of an existing ETL that selects from datasets, dealing with deletes?

Thanks,
Ben

view message
ETL dataset with a large IN filter?
(5 responses) Ben Bimber 2018-04-03 16:02

I'd like to ETL a large dataset from one server to a second labkey server. The source table has data on a large # of subjects; however, I only want data from about 1000-2000 of them, and that list changes regularly. That whitelist of subjects only exists on the destination server (in a table). Is there a good way to implement this type of filter in the ETL? I am assuming an IN clause (which is basically what the ETL presents) would be a problem with this many IDs, but I havent formally tested that. I am considering requesting the entire dataset in the ETL, and then trying to add some kind of java transform code that would run on the destination server to skip over rows I dont want (probably not the most efficient, but maybe better than nothing).

The best case would probably be to make that whitelist of IDs available to the source server in a database table, so I could write a SQL join query, and then request this query unfiltered on my destination server. It would in theory be easy enough to make an ETL in the opposite direction to sync the whitelist of IDs from the destination server to the source server; however, for a variety of reasons I cant practically deploy code to the source server without a lot of hoops. My understanding is that I'd need code in order to add this ETL, is that right?

Is there another way to think about this problem that I'm missing?

Thanks

view message
Null pointer exception from js call LABKEY.Message.sendMessage
(2 responses) Will Holtz 2018-03-29 18:46

Hi,

In Labkey v17.1 (and earlier) I could send emails from a dummy user (announce@example.com) in my Labkey instance using the following code:

 <script type="text/javascript">
         function errorHandler(errorInfo, responseObj)
         {
             LABKEY.Utils.displayAjaxErrorResponse(responseObj, errorInfo);
         }
        function onSuccess(result)
         {
             alert('Message sent successfully.');
         }
        LABKEY.Message.sendMessage({
             msgFrom: 'announce@example.com',
             msgSubject: 'Testing email API...',
             msgRecipients: [
                 LABKEY.Message.createRecipient(LABKEY.Message.recipientType.to, 'realuser@example.com'),
             ],
             msgContent: [
                 LABKEY.Message.createMsgContent(LABKEY.Message.msgType.plain, 'This is a test message')
             ],
             success: onSuccess,
             failure: errorHandler,
         });
         </script>

In v18.1 this results in the user viewing the page with the js code to get the error 'An error occurred trying to load: java.lang.NullPointerException' and the following appears in the logs:

WARN SendMessageAction 2018-03-29 18:03:42,943 http-bio-443-exec-8 : The user: announce@example.com is either disabled or has never logged in and has been omitted.
WARN MailHelper 2018-03-29 18:03:43,050 http-bio-443-exec-8 : Exception sending email; check your SMTP configuration in labkey.xml
from: null
to: realuser@example.com
subject: Testing email API...
java.lang.NullPointerException
at org.labkey.api.util.MailHelper.getAddressStr(MailHelper.java:242)
at org.labkey.api.util.MailHelper.addAuditEvent(MailHelper.java:226)
at org.labkey.api.util.MailHelper.send(MailHelper.java:200)
at org.labkey.announcements.SendMessageAction.execute(SendMessageAction.java:132)
at org.labkey.announcements.SendMessageAction.execute(SendMessageAction.java:58)
at org.labkey.api.action.ApiAction.handlePost(ApiAction.java:180)
at org.labkey.api.action.ApiAction.handleRequest(ApiAction.java:125)
at org.labkey.api.action.BaseViewAction.handleRequest(BaseViewAction.java:177)
at org.labkey.api.action.SpringActionController.handleRequest(SpringActionController.java:415)
at org.labkey.api.module.DefaultModule.dispatch(DefaultModule.java:1231)
at org.labkey.api.view.ViewServlet._service(ViewServlet.java:205)
at org.labkey.api.view.ViewServlet.service(ViewServlet.java:132)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.labkey.api.data.TransactionFilter.doFilter(TransactionFilter.java:38)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.labkey.api.module.ModuleLoader.doFilter(ModuleLoader.java:1138)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.labkey.api.security.AuthFilter.doFilter(AuthFilter.java:214)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.labkey.core.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:118)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:522)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1095)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:672)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:279)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)

Because my dummy user has never logged into my Labkey instance, getEmail(from) in server/modules/announcements/src/org/labkey/announcements/SendMessageAction.java line 124 returns null in v18.1 and this is the root of the problem. I logged into my dummy account and this resolved the issue for me, but I thought I'd report it in case there is a desire to clean up the error handling here.

-Will

view message
Difficulties adding my own CSS to style LabKey interface
(2 responses) olnerdybastid 2018-03-29 12:19

I have a few questions stemming from my recent attempt to add CSS to an interface we're building in LabKey. In case it's relevant for troubleshooting, the UI I'm trying to build is located within a wiki web part. I've tried the following 3 approaches so far without success, and am curious to know why these methods aren't working (or in the case of the first, isn't working as expected).

  1. First, I followed the documentation here (https://www.labkey.org/Documentation/wiki-page.view?name=customizeLook#resources) and uploaded my own test CSS file. This method worked, but it seems as though LabKey cached the first CSS file I uploaded, and it now loads the same one each time--even after I deleted the file and tried to upload a new one in its place. Accessing our LabKey server (and after restarting Tomcat) from a different browser still loads that same CSS.

  2. Next, I put my CSS file (lkmain.css) in the same folder as the other out-of-the-box LabKey stylesheets (for me, the full path is /Applications/labkey/labkeywebapp/core/css) and added the following line to my javascript:

'LABKEY.requiresCss('core/css/lkmain.css');'

This throws a 404 error each time I load the page saying that resource can't be found. If I point to one of the predefined LabKey CSS files in that folder I don't see the same error.

  1. Finally I tried to link to my CSS (located in the same path as before) from within the <head> of my HTML. No error gets thrown this time, but the CSS I linked to isn't being loaded (as verified by looking at the Resources tab of the Developer console in Safari).

Any suggestions/explanation of why these methods aren't working would be most appreciated.

view message
Stored Procedure
(4 responses) marcia hon 2018-03-22 06:54

Hello,

I have a stored procedure that is supposed to work with assays.

How do I write the table in the stored procedure. I have tried: "FROM assay.General.Genotyping_Original.Data instrument;" but this gives me an error.

How can I, inside a stored procedures, connect to an existing assay?

Thanks,
Marcia

view message
Running two distinct sites off one LabKey Server instance?
(3 responses) Ben Bimber 2018-03-21 09:14

I have talked with some of you about this, but I wanted to revisit something we're considering. We have a couple smaller grants that would like a public facing LabKey site. These will be low use, and there is a lot of extra cost and time from paying for distinct VMs, configuring LK instances, etc. specific to each project. It would be really nice if we could host all of these off one LK instance, possibly adding/removing projects over time. Obviously I can make separate projects for each one; however, it would be highly preferable to make each one look and behave to users as though it was the only thing being hosted off that LK instance. A single site with different project doesnt behave like that at all. My base assumption is that each 'site' is actually a project within LabKey.

Requirements would be:

  • dedicated URL for each project (i.e. https://myproject.ohsu.edu, not https://labkey.ohsu.edu/myproject/). I think I can do some configuration in httpd to shuttle traffic from that project-specific URL to the LK project URL, but have not tried that. i dont know if there will be gotchas from the different host names. it would be nice to keep the URL in the browser with that project-specific URL, as opposed to doing a one-time redirect and then using generic URLs.

  • whenever people access their project, all site UI should look as though it is specific to their "site". for example, each project should have their own logo, and maybe color scheme. i believe that's possible to configure project-level already

  • this does mean regular users should never get UI that directs them to anything outside their project. this is the trickiest one. a lot of UI in LabKey directs you to /home, and because there is one /home shared across all these 'sites', it would need to be non-branded. it's ideally a never accessed folder.

Has anyone attempted to thought about this, and are there any pitfalls or problems I'm not thinking about?

view message
Specimen Archive frozen_time not visible
(1 response) Edward 2018-03-21 09:02

Hi,

I am uploading specimen archive to labkey where it has values for forzen_time specified in the tsv file. However, in the grid view labkey does not show the frozen_time column. Also the column is not visible when I modify the grid view and so I cannot include it. At the backend, the column is present in the database and has values added in it. Is this a bug?? or am I missing something?

Also, is it possible to add user-defined fields to the specimen event? I added (for example) a field "frozen_date" to the specimen event and it is reflected in the database but the grid view does not show me this new field.

view message
File Transfers Between Projects
(2 responses) slatour 2018-03-20 08:24

Hi,
I am trying to transfer files between projects on LabKey without having to export the entire file repository of a project and then individually deleting folders once they are imported to the desired project. Is it possible to select one folder/file at a time to transfer between repositories in projects? As of right now I am only able to export a project and select files as an export option and have ALL files be exported into a repository within my desired new project. Unfortunately I only want a few select files not the entire contents of a repository.

Any help would be appreciated.

Thanks,

-Sara

view message
API insertRows to add samples fails with error 'Name expression required'
(5 responses) olnerdybastid 2018-03-16 16:00

I am trying to use the LabKey API to programatically insert new samples, but keep getting an error stating 'Name expression required' (though I am providing a value for 'Name'). I've verified that my API calls to select rows are good in the same table I want to insert into, so it seems to me like the problem is with the structure of the data I'm trying to pass to LabKey, and probably stems from a misunderstanding on my part of how the sample schemas are structured. I've reproduced a minimal example below using R and the HIV Study tutorial (I get the same error using JS):

base = 'http://localhost:8080/labkey'
folder = '/Tutorials/HIV Study'
schema = 'samples'
query = 'Study Specimens'

session <- getSession(baseUrl=base, folderPath=folder)

# Get samples from the tutorial HIV study -- this works, returns 496 rows
samples = labkey.selectRows(baseUrl=base, folderPath=folder,schemaName=schema, queryName=query)

# Now take the first row, change the name, and try to upload this as a 'new' sample 
sampleToUpload = samples[1,]
sampleToUpload[1,'Name'] = 'thisIsMySample'

# Doesn't work, returns 'HTTP request was unsuccessful. Status code = 400, Error message = Name expression required'
labkey.insertRows(baseUrl=base, folderPath=folder,schemaName=schema, queryName=query, sampleToUpload)
view message
Data Transform Webpart
(2 responses) marcia hon 2018-03-16 06:27

Hello,

I have created an ETL module based on your tutorials. It works fine.

However, how do I add a "Data Transforms" web part? Is there some sort of configuration that I must do before?

Thanks,
Marcia

view message
build fails with cannot find symbol StringUtils
(2 responses) Will Holtz 2018-03-14 13:16

I have been trying to build r56184 (I think this is the 17.3 release) and it was failing with:

/Users/wholtz/code/labkey/trunk/server/modules/query/src/org/labkey/query/controllers/AbstractExternalSchemaForm.java:18: error: package org.apache.commons.lang does not exist
import org.apache.commons.lang.StringUtils;
                              ^
/Users/wholtz/code/labkey/trunk/server/modules/query/src/org/labkey/query/controllers/AbstractExternalSchemaForm.java:49: error: cannot find symbol
        if (StringUtils.isBlank(bean.getUserSchemaName()))
            ^
  symbol:   variable StringUtils ```

It appears that org.apache.commons:commons-lang changed to org.apache.commons:commons-lang3 and that the dependency on org.apache.commons:commons-lang is no longer declared. 
``` ./gradlew :server:modules:core:dependencies | grep lang ``` only gives command-lang3:3.5 results. I'm pretty clueless about setting up dependencies in gradle, but I edited server/modules/query/build.gradle to include ``` compile group: 'commons-lang', name: 'commons-lang', version: '2.6' ``` within the dependencies block and was then able to get the build to successfully complete. There may be better ways to do this.

I solved my problem but wanted to post here in case someone else hits this or if an improvement needs to be made to the dependencies definitions.

-Will
view message
Using triggers to prevent insert
(2 responses) Sev 2018-03-13 15:32

Hello

I would like to restrict the insert of records into a dataset based on what values have historically been used for a given field. In particular, I am trying to prevent reuse of patient ids in the Demographics dataset. For example:

In the demographics dataset, a patient is assigned an id of 12345. Some time later, the patient is removed from the study and deleted from the demographics dataset. The patient id 12345 is no longer in the demographics dataset, but we want to prevent this id from being used again.

I have some postgres triggers setup to create a patient id history table, and a trigger applied to the demographics dataset to check against the history table when a new row is inserted. If the patient ID is found in the history table the insert is aborted, but I am unable to bubble up an error message to the UI to alert the user what has gone wrong.

I was looking into the trigger scripts documentation and I've been able to setup a beforeInsert trigger. I see how I can use the Labkey.Query API to check against the history table using the "row" parameter of the beforeInsert function, but I don't see a way to prevent the insert, or to provide a message to the user in the UI.

Is there a way I can prevent a record from being inserted as I've described and alert the user in the case that they are re-using an old patient id?

Thanks!

Sev

view message
RLabkey API Performance
(1 response) marcia hon 2018-03-08 08:27
Hello,

From our experience, RLabkey API Performance is very slow.

We have a query which uploading directly to Postgres takes 5 minutes... whereas via RLabkey API it takes days.

Please let us know what is wrong and how to fix this.

Thanks
Marcia
view message
Speed of Queries in Labkey
(1 response) marcia hon 2018-03-08 08:05
Hello,

Is there some configuration settings that I can modify in order to make my Labkey queries run faster?

I'm having a very hard time with queries that do many calculations and also pivots.

One query took up to 30 minutes to return results.

Thanks,
Marcia
view message
Grouping in Pivot Query
(1 response) marcia hon 2018-03-07 10:20
Hello,

I have a query, that requires a pivot. However, I am unable to get the group by to work appropriately. My query is:
...
'CYP1A2_rs762551'
)
AND D.LabID<100
GROUP BY D.SeriesName, D.LabID, D.CollaboratorSampleID, D.Sex, G.Locus, G.Allele1, G.Allele2

PIVOT Alleles BY Locus

ORDER BY SeriesName, LabID, CollaboratorSampleID, Sex

...


I would like Group by to work for : D.SeriesName, D.LabID, D.CollaboratorSampleID, D.Sex

Not to include Locus, and Allele1/2

I look forward for your insight.
view message
Queries in Postgres
(3 responses) marcia hon 2018-03-07 06:52
Hello,

I created queries in Labkey.

Where in Postgres are these queries located?

Thanks,
Marcia
view message
"data_2" error
(1 response) marcia hon 2018-03-05 11:08
Hello,

How do I fix this error:

View has errors
org.postgresql.util.PSQLException: ERROR: table name "data_2" specified more than once


My script is the following:

    SELECT G1.SeriesName AS SeriesName, G1.Locus AS Locus, COUNT(DISTINCT G1.LabID) AS ConflictingGenotypes
    FROM NRG01_CMH_assay_Genotyping AS G1
        INNER JOIN NRG01_CMH_assay_Genotyping AS G2
                ON G1.ParticipantID = G2.ParticipantID
                   AND G1.Locus = G2.Locus
                   AND (G1.Allele1 <> G2.Allele1 OR G1.Allele2 <> G2.Allele2)
       GROUP BY G1.SeriesName, G1.Locus
view message
Minimum necessary permissions on SQLServer?
(5 responses) Ben Bimber 2018-03-02 09:21
For as far as I remember, the LabKey guidance for installing on SQLServer has bee that the labkey database account should be syadmin or equivalent. While this obviously grants the LK application permission to do whatever it needs, it really is far in excess of what it really ought to be. LabKey does install the CLR functions and create a database; however, it doesnt manger users/permissions or things like this. In an increasingly security conscious environment, having that additional sysadmin account, even if it's a service account, is less defensible and we're getting from DBA pushback. In a scenario in which a single Sqlserver instance hosts multiple LK servers (each potentially sub-admined by different people), that global SQL permission is also less appealing.

Have you considered whether you could issue recommendations/guidance on a more targeted permission set for SQLServer permission set that would give the LK app permission to do what it needs, without opening up everything?

Thanks in advance.
view message
How to create a new Schema
(2 responses) marcia hon 2018-03-02 08:05
Hello,

How do I create a new schema?

Thanks,
Marcia
view message
Remove shared folder
(2 responses) Edward 2018-03-01 08:09
Hi all,

Is there any way to delete the "shared" folder? I tried to use folder management but there I did not see any delete option for deleting the shared folder. However, I do see delete button for other folders. Anyone with a good suggestion?

Thanks in advance!
view message
R-Report - Saving
(1 response) marcia hon 2018-02-27 08:33
I have the following:

```{r setup, echo=FALSE, cache=TRUE, results='hide', warning=FALSE, comment=FALSE, warning=FALSE, message=FALSE, encoding = 'UTF-8',comments = ""}
knitr::opts_chunk$set(echo=FALSE, cache=TRUE, comment=FALSE, comments = "", warning=FALSE, eval = TRUE, encoding = "native.enc")
knitr::pandoc('script.Rmd', format='html')
```

```{r, echo=FALSE, cache=TRUE, results='hide', warning=FALSE, comment=FALSE, warning=FALSE, message=FALSE, encoding = 'UTF-8',comments = ""}
currentdir <- getwd()
assign("html_filename", paste(currentdir, "/script.html", sep=""))
assign("out_filename", paste("/directory/@files/Reports/Results/Report_", format(Sys.time(), "%Y-%m-%d_%H:%M:%S"), ".html", sep=""))
file.copy(from = html_filename, to = out_filename)
system("rm script.* *.tsv")
```

This code sometimes saves the file and sometimes does not save the file. Is there a way to force it to save always?

Thanks,
Marcia
view message
WebPart
(1 response) marcia hon 2018-02-27 08:22
Hello,

I would like to create a WebPart that contains links to queries.

For example:

1. query01
2. query02
3. query03
4. query04
5. query05

Please let me know how to accomplish this.

Thanks,
Marcia
view message
Specimens dataset customization
(3 responses) Edward 2018-02-27 06:58
Hi,

I have a specimen dataset with the following fields:

global_unique_specimen_id   
specimen_number   
draw_timestamp   
visit_value   
ptid   
Sample_Taken_Status   
Sample_Type   
Sample_Arrive_Date   
Sample_Freeze_Date   
Sample_Freeze_Time   
Location

If I use the default labkey specimen template then I labkey does not add the last 6 fields. Is there a way to define these as specimen columns and then whenever I upload specimen data, the data is mapped on to these columns??
view message
SQL Query with Parameters
(4 responses) marcia hon 2018-02-23 10:32
Hello,

I wish to create an SQL query that uses parameters.

These parameters I would like to use in the "IN" command.

For example:

WHERE values IN parameters

How do I do this? If I enter something like this in parameters it does not work: value1, value2, value3

Thanks,
Marcia
view message
Length of a Column Exceeds 4000 characters
(1 response) A. Tyrell 2018-02-21 13:04
Hi,

I would like to import data in either a list or a dataset but I am running into an issue where I cannot import a column if its length exceeds 4000 characters.
I changed the data type from Text to Memo as well.

Thanks,

Alicia
view message
Creating a Query with data from different schemas
(1 response) marcia hon 2018-02-21 09:22
Hello,

Is it possible to do a query with data from different schemas? For example, one is from Assay and the other is from Study.

How do you do this?

Thanks,
Marcia
view message
R Report - How to use variables
(1 response) marcia hon 2018-02-20 11:59
Hello,

I was able to do this:

participant <- labkey.data$participantid;
visit <- labkey.data$visitid;

My question is how do I use these variables in html? For example:

<b>participant[1]</b>

??

Thanks,
Marcia
view message
Printing R Report
(3 responses) marcia hon 2018-02-20 11:30
Hello,

I have created an R Report using HTML.

How do I print this report?

Thanks,
Marcia
view message
Image for R Report
(1 response) marcia hon 2018-02-20 07:55
Hello,

I would like to add a logo to my R Report that uses HTML.

I have tried the following:

<center>
<img src="/external/rprshnas01/nip/rdrshlabkeyv/ETL_TEST/ETL_Workpace/@files/report_logo.jpg"/>
</center>

I think the problem is what directory I should point to for the logo?

I look forward to your input.

Thanks,
Marcia
view message
Transform scripts for preprocessing uploads to assay modules
(1 response) zromer 2018-02-19 21:30
If I have data that is being uploaded to an assay module which doesn't match the format required by the module can I use a transform script to preprocess / transform the data into the required format? This was my original understanding of what they were for but I'm having trouble getting the transform scripts to run on data requiring transformation before upload, while they seem to run fine on data which is already in the right shape.
view message
ETL to Dataset
(4 responses) marcia hon 2018-02-16 09:58
Hello,

I have an ETL module. It gets data from an assay and then uploads it to a dataset.

The problem is that it is making the combination of ParticipantID and VisitID as a key. Therefore, I cannot upload more of this combination.

My understanding, is that this should be possible.

A solution that I'm thinking of, is to make all columns together as the primary key. Please could you let me know how to specify what belongs to a primary key?

I look forward to your feedback.

Thanks,
Marcia
view message
Remove unwanted menu items
(1 response) Edward 2018-02-15 08:06
Hi,

I am using Labkey within an organization for storing data and want to have a strict control over different menu items.
Is there a way to remove unwanted menu items such as, Give UI Feedback and/or Labkey Documentation?

Regards,
Edward
view message
Problems with Python transformation script
(1 response) zromer 2018-02-14 12:31
Hello,

I have a Python engine set up on my Atlas instance with $(runInfo) configured as the Program Command (screenshot of engine configuration attached). I also have a python transformation script added to a particular assay module. I'd like this transformation script to take the assay data being uploaded by the user before Atlas tries to match its columns to the assay data result set definition, do some preprocessing on it, then write the file out to be received by the assay module. I'm having two issues with this.

1) How do I ensure my transformation script is getting called? I'm currently trying to have my script send and email and write out a test file to the local execution directory but neither seems to be happening upon upload to the Assay Module my transformation script is a part of.

2) If I understand correctly, the Program Command with $(runInfo) will pass a runProperties.tsv file to my python script as a command-line parameter which I will then need to read in order to get the runDataUploaded property which contains the file path of the data uploaded by the user. Then I will do my transformation on the data after I've read it in. Then... outputting the transformation gets fuzzy to me.

Do I then write the transformed file out and replace the runDataUploadedFile attribute with the transformed file path, then write out the runProperties.tsv file to... where? Or do I replace the transformedRunPropertiesFile attribute with runProperties.tsv after I've updated the runDataUploadedFile attribute with the transformed file path? Or something else? It is generally unclear to me how the program picks up the output of a transformation script.
 python_engine.PNG 
view message
Python Engine
(2 responses) marcia hon 2018-02-12 12:47
Hello,

Please help me complete the Python Engine:

I have the following:

     Language - python
     Language Version – 2.7.12
     File Extensions - .py
     Program Path - /usr/bin/python


What do I put for:
     Program Command
     Output File Name

?
view message
Labkey github asks for password
(1 response) Edward 2018-02-12 10:28
Hi,

I am trying to download/clone labkey biologics module from the official github of labkey but it asks me for username and password. I tried cpas both as username and password but it does not work. Do anyone know what username and password I have to use??
view message
Conditional Validation
(1 response) marcia hon 2018-02-12 07:40
Hello,

We have a situation where:

1. If DOBUnknown is checked off, then DOB should be empty
2. And vs versa, if DOB has a values, DOBUnknown should be empty

How could we enforce these conditions?

Thanks,
Marcia
view message
Labkey Biologics and Labkey Server
(1 response) Edward 2018-02-12 04:34
Hi,

I would like to use both Labkey server and Labkey biologics on the same computer. Is it possible?

I am asking this question because the instruction for setting labkey biologics states that labkey biologics can be enabled in labkey server by re-building the code for labkey server (https://www.labkey.org/Documentation/wiki-page.view?name=buildingBiologics&_docid=wiki%3A16340131-7af1-1035-b86e-fe851e083e48).

I would like to keep both labkey server and labkey bilogics because of the additional features by biologics. Do anybody know if it is possible?
view message
RLabkey - Upload Taking a Long Time
(2 responses) marcia hon 2018-02-09 05:28
Hello,

We are trying to automate the process of uploading into Labkey.

We are trying to use RLabkey library to insert.

There are 50,000,000 records and we kept it running. Still after one week it is still uploading.

I tried uploading via the "Bulk Upload GUI" option, and it takes 25 minutes.

Please let us know how to get RLabkey to upload in a good time as we would like to automate this process.

Thanks,
Marcia
view message
Uploading CSV to List
(2 responses) marcia hon 2018-02-08 08:02
Hello,

I uploaded a CSV file via Admin>Site>AdminConsole>Pipeline .

How can I now push this CSV file to a List?

The reason I am asking is that we have huge files that need to be uploaded and I'm trying to find ways to upload the files.

Thanks,
Marcia
view message
Filtering View/Report via URL
(1 response) Matt V 2018-02-05 16:29
I have a DB constructed view combining a few datasets that other pages within a labkey instance link to. I'd like to pass a filter arg via the URL similar to how it's done with datasets (mentioned on https://www.labkey.org/Documentation/wiki-page.view?name=sortingData#url).

I've tried just a basic sort via reports.sort=ParticipantID, report.sort=ParticipantID, dataset.sort=ParticipantID, etc. However, sort doesn't seem to work with this view/repore let alone filtering. If I do the same on the dataset's URL it works. Is there a way to accomplish this for a view/report?
view message
Transformation Script
(2 responses) marcia hon 2018-02-05 09:45
Hello,

Is there such a thing as a Transformation script for datasets? I have done it with assays, but not datasets.
view message

Welcome to the LabKey Developer Forum. This forum is for questions about development with LabKey Server

If you have questions regarding general use of LabKey Server, please post to the General Server Forum.

If you have questions regarding installation and configuration of LabKey Server, please post to the Installation Forum.

Posting Questions

When you post a question, please include the following information:

  • Your operating system.
  • Web browser.
  • Version number of LabKey Server.
  • A detailed description of your problem or question, including instructions for reproducing your issue.
  • Error information. Please attach log files to your message, rather than pasting in long text. Beginning with release 17.3, error pages and log files will include an Error Code. Please include this code in your message.

Additional Resources

User Account

In order to post to the community forum, you'll need to register for a user account.  If you already have an account but have forgotten your password, you can reset your password using the link on the Sign in page.

                                                                                 

Welcome to the LabKey Developer Forum. This forum is for questions about development with LabKey Server

If you have questions regarding general use of LabKey Server, please post to the General Server Forum.

If you have questions regarding installation and configuration of LabKey Server, please post to the Installation Forum.

Posting Questions

When you post a question, please include the following information:

  • Your operating system.
  • Web browser.
  • Version number of LabKey Server.
  • A detailed description of your problem or question, including instructions for reproducing your issue.
  • Error information. Please attach log files to your message, rather than pasting in long text. Beginning with release 17.3, error pages and log files will include an Error Code. Please include this code in your message.

Additional Resources

User Account

In order to post to the community forum, you'll need to register for a user account.  If you already have an account but have forgotten your password, you can reset your password using the link on the Sign in page.